To follow this blog by email, give your address here...

Monday, June 13, 2016

Why the Good Guys Will Generally Win

This is a brief post, typed hastily, but it captures some ideas that have been germinating in my mind for quite some time.

It's definitely a “stoned college sophomore” type of post (though at the moment of writing it I am neither stoned nor a college sophomore), playing fast and loose with a bunch of highly subtle and important concepts, in a fun and preliminary way.   So take it for what it is … or don't ...

The topic is what some would call “Universal Ethics” – i.e., what are the ethics intrinsic in the dynamics of the universe itself?   Various people have attempted to argue, in various ways, that the universe tends toward Good rather toward Evil in various senses, but none of the arguments I've read have quite resonated with me (perhaps due to shortcomings in the arguments, or perhaps just to peculiarities in my own perspective, and my need to think things through for myself and understand them in my own way rather than just eating other peoples' arguments).  

In any case, here I will present a sketch of my own argument that the universe will generally tend toward Good rather than Evil.

The argument I will give here is quite vague and hand-wavy, at least in the form in which it's presented.   Going out on a long creaky limb, I'll venture that it could potentially be turned into a formal mathematical proof, if one set up the framework right and did an awful lot of work.  This would be interesting, but of course it wouldn't make the argument definitively compelling in a philosophical sense (though it might well make it more compelling by adding detail and flavor in various useful ways).   Ultimately arguments about ethics and the universe we live in can't be settled by mathematics alone, because mathematics only has practical meaning when it is used in conjunction with appropriate bridging assumptions (e.g. physics, or mathematical metaphysics).  

I will start with three values that I have identified as core to the Cosmist value system (the variety of human value system that appeals to me most, and that I think has the greatest growth and survival path for the future): Joy, Growth and Choice.   For discussion of why these values seem critical to me, see the relevant sections of my books The Hidden Pattern (free pdf here) and A Cosmist Manifesto (free pdf here).   In brief, and in a very fuzzy/humanistic way:
  • Joy is joy!  That is basic goodness.  Who can argue with joy?  In the context of complex cognitive systems like people or AGIs, joy is closely allied with the feeling of a system getting what it wants and expects; which of course is often achieved via the system adjusting what it wants and expects judiciously.
  • Growth is important too, because with just joy and no growth, one might just have a universe comprising one big fuzzy orgasm.   Of course, from the point of view of the meta-cosmic orgasm itself, the meta-cosmic orgasm is No Problem At All.   But from my point of view as a complex cognitive system, I'm not satisfied with that, and I want to see new patterns keep getting created.  Growth is creativity, it's evolution, it's life.
  • Choice is individuality.   With just joy and growth, we might just have some sort of universe-wide pattern-generating fractal – with no individual mind-systems pursuing their own goals and refining their own world-views.   But as a semi-autonomous human agent, the ongoing existing of individuals making choices feels important to me (though note the complexity of the concept of choice: some sort of “natural autonomy” is more interesting to think about than incoherent notions of “free will”).

Of course, these three values don't summarize the whole of human values, nor the whole of my own personal value system.  Furthermore, these three values – as I mean them – are complex constructs whose deep definition and meaning depends on a huge amount of human culture.  But nevertheless, I think these three values are symbolic of thought/feeling-networks that are very core to human values and encompass a large percentage of human values.   In terms of “glocal memory”, I think these three values are “keys” corresponding to distributed attractors in the human collective mind, and that these distributed attractors approximate the essential aspects of human values interestingly well (though far from completely).

Given this background: One thing I want to present here is a basic, simple argument why Joy and Choice promote Growth very effectively.   There is nothing profoundly new here, I am just summarizing some obvious ideas in a slightly different way than I've seen before.  

If one can show that Joy and Choice promote Growth very effectively (and in particular, that they promote it more effectively than suffering and de-individuation), then to show that the universe intrinsically promotes Joy, Choice and Growth, one basically only needs to show that the universe promotes Growth.   Because from an evolutionary view, if one looks at the universe as a large system containing various subsystems, then if
  • Joy and Choice promote Growth better than suffering and de-individuation do
  • the universe promotes Growth

it follows that statistically, subsystems with Joy and Choice will tend to prevail.  

But the fact that the universe promotes Growth, in the sense that it rewards growing systems via differentially favoring their existence, follows fairly directly from the finitude of the universe's resources.   In a universe with highly limited resources, systems that can grow a lot will grab up more of the resources, shutting out systems that can't grow that much.   This sort of phenomenon is seen in evolving ecologies of every sort.

But why do Joy and Choice promote Growth so effectively, so much better than their opposites?   This is also easy enough to see:
  1. Joy promotes greater generosity, which promotes greater collective intelligence.   A system that is getting what it wants and feels it needs, is more likely to share with other systems.   Thus, a collection of joyful systems is more likely to share resources and information amongst each other copiously, than a collection of less joyful systems.   Less joyful systems, that are not getting what they need, are more likely to conserve what they have for themselves and avoid sharing with others, as they will be occupied with scheming to get what they need.   But greater generosity is the key to superrationality, which is the key to collective intelligence.   Collections of joyful agents are more likely to display powerful collective intelligence.   (There is some yet-to-be-unraveled mathematics underlying generosity and superrationality; the incomplete and awkward thoughts I wrote down in this rough draft a few years ago may provide a few inklings in this direction).
  2. Choice promotes greater diversity of pattern generation.   The core reasons for this are subtle, and in the end boil down to the finitude of energetic and informational resources once again.   The nature of pattern-space is that patterns are easier to detect if one restricts attention to relatively local regions of pattern-space (i.e. regions consisting of patterns fairly similar to each other).   To recognize patterns across a very broad region of pattern-space, requires much more intelligence, i.e. much more energetic and computational resources. Thus if one wants to recognize a large number of patterns, it is generally better to have a collection of pattern-recognition processes each looking in some relatively local region of pattern-space, and focusing on that region with inductive biases appropriate to that region. There is some as-yet-unraveled mathematics underlying this point, which I predict will be very interesting to unfold over the next decades.   (This closely relates to some of the ideas underlying OpenCog's MOSES component, which carries out automated program learning via a set of small evolving populations of programs called “demes”, each of which seeks to learn programs in a certain semi-local region of program space.  The reason MOSES works this way is that certain sorts of patterns (“syntactic-semantic correspondences between programs”) are easier to recognize in a small region of program space than in a large one. But I think the computer-science observations underlying MOSES actually represent more general principles.)

So there we have it.   To re-cap:
  1. Joyful systems will tend to grow better than non-joyful systems, because joy fosters generosity which fosters collective intelligence driving superrationality
  2. Choiceful systems will tend to grow better than non-choiceful systems, because choice fosters diverse pattern generations
  3. Growing systems will tend to prevail over non-growing ones, because they will tend to grab up the universe's finite resources more rapidly

So from the assumption of the finitude of the universe, we arrive at the conclusion that systems embodying Joy, Growth and Choice will be more likely to prevail.

Of course, even if all this is correct, this is purely a statistical argument, and doesn't show that Good Things will prevail in any particular situation.   Nor is it the kind of statistical argument that is likely to yield accurate calculations of the specific odds of Good prevailing in any particular real-world situation.  

But as a piece of speculative philosophy, I find this direction intriguing, and – in the random odd moments I find to devote to such things – I hope to refine this line of thinking in more detail in coming years.  

Among other things, this argument provides a slightly different angle on the Problem of Evil – which I often consider in the form “Why does an infinite universe want to divide itself into finite subsystems, when the limitation of resources within these finite subsystems causes so many internal problems from the perspective of entities within these subsystems?”  ….  

The angle suggested here is: Finite universes exist because, within them, there is a dynamic via which Good (i.e. Joy, Growth and Choice) will statistically prevail.  

Which is a fancy way of saying what I said much more simply in A Cosmist Manifesto: Separation exists so that there can be the joy of overcoming separation, which is love.   And this of course, is not an original notion at all.   But the unraveling of this notion in terms of evolutionary dynamics, pattern space, superrationality, and so forth, is fairly new within our shard of the meta-cosmos, and still mostly to be unfolded.

As an exercise for the reader, it may be interesting to modify the above long winding choo-choo train of thought into an argument why happy societies with ongoing progress and individual liberty are ultimately going to prevail on average.   Good news for those concerned about society and politics!

Sunday, June 05, 2016

The film “Machine of Human Dreams” – some comments by the human dreamer



In which the primary subject of a recent documentary film rambles on a bit about the events the documentary covers, and some things that it leaves out...

These are amazing times for those of us who are passionate about AI and robotics. Finally, at ever so long last, the ideas and visions we've been talking about and working toward for decades, are getting embraced by the mainstream! Concepts that would get you laughed out of university departments or corporate research labs just 10 or 15 years ago, are now being adopted as research priorities by governments and major corporations. Believing that AIs with general intelligence at the human level and beyond may well get created during our lifetimes, no longer makes you a certifiable member of the lunatic fringe!

One practical consequence of this shift in the zeitgeist is that funding for advanced AI R&D is now less difficult to come by. It's still not easy – funding is always competitive, and the dynamics of various funding sources remains complex. But things are much better than they were a decade or two ago, and this shows not only in big-time funding events like Google buying Deep Mind for a half billion dollars or Elon Musk and friends soft-committing $1B to OpenAI, but also in smaller ways like the OpenCog project I lead getting more donations and corporate backing. After a number of years of slow progress throttled by resource limitations, we are starting to move faster.

Another consequence of the increasing enthusiasm for AI is growing media attention. The number of calls and emails I get from journalists these days is remarkable. And this summer the second documentary film about my AI and robotics work is coming out – “Machine of Human Dreams”, by Roy Cohen and Roast Beef Productions.

The first film made about my work was Raj Dye's avant-garde documentary “Singularity or Bust”, which covered my collaboration with Hugo de Garis in China in 2009 (and is available for free now on YouTube). Raj's film won Best Documentary at the LA Film Festival of Hollywood, and I like it very much. The style is a bit home-movie-ish at times, but it works; the editing and direction are very thoughtful, and a certain slice of my life and work is captured with artistry and accuracy.

Machine of Human Dreams covers bits and pieces of my AI and robotics work in Hong Kong and Addis Ababa during the period 2013-2015. It also covers portions of my earlier life and career.

Not too surprisingly, I definitely recommend you to watch the film. I particularly like the parts of the movie covering my team's recent work in Hong Kong and Addis – I think these are excitingly shot and directed, and they show aspects of our recent robotics tinkering that there's no other way to get a visual look at. 

The footage of RoboSapien robots (that Ruiting and I brought from Hong Kong in our suitcases) dancing in the rugged streets of Addis Ababa packs a sociopolitical wallop along with the techno-pizazz … and David Hanson's Sophia robot, showcased near the end of the film, is just so frickin' beautiful and evocative.... 

The various futuristic discourses and diatribes the film captures me giving are mostly well selected, and get across my broad high-level vision for AGI fairly nicely. There's not much footage of me explaining the meat of my AI work, but then there are numerous YouTube videos of me giving such explanations already available, so anyone who cares can find such material.

Trying to cover so much stuff in just an hour or so, it's inevitable that the film leaves a bunch of important things out … and I have to admit that some of the choices Roy made regarding what to include and what (and who) to omit, were pretty different than the choices I would have made given all the footage he gathered of myself, my team, and my friends and family.  With this in mind, I have written this probably over-long blog post in order to comment on the events the film covers, and also to highlight some of the things the film leaves out, which I think are nonetheless fairly critical to understanding the events and individuals that the film presents.

So – my hope is that, for someone who has seen the film, this post will fill in some of the “missing links” and make the whole story clearer.  

If you haven't seen the film, there is probably still some useful and entertaining info in this post; but the choice of specific topics to discuss here is heavily driven by the various scenes shown in Machine of Human Dreams, so it will definitely make more sense if you've seen the movie. 

Also, I suppose that alongside their potential interest to AGI geeks, my comments here may also be of interest to anyone curious about the general nature of the complex relationship between documentary films and actual reality....

The Film Originates...

I first met Roy Cohen at the Global Future 2045 conference in New York in 2013, where David Hanson was planning to demonstrate one of his robots. The robot demo at the conference wound up not coming together for various logistical reasons, but David was there at the conference, and was there afterwards as well holed up in a hotel room getting his Philip K. Dick robot ready for a TV interview. Roy was trying to track down David, but David was busy fixing up his robot. But I had a little more free time. 

Roy seemed like a smart guy – he had a neuroscience background and a broad understanding of technology, and he was getting into film-making. He wanted to do a documentary project. I suggested he should make a documentary on the collaboration David Hanson and I were starting, aimed at applying my open-source OpenCog AGI software to control David's amazing humanoid robots.

We talked more and the concept grew on Roy. He scrabbled together some funding from Israel to do some preliminary shooting. He came to Hong Kong and got some footage of our robot lab; and he came to Ethiopia to film the team at iCog Labs, the AI/robotics development firm I co-founded there, that collaborates with OpenCog and Hanson Robotics. The preliminary footage he shot enabled him to gather enough funds to complete the film.

I liked the idea of Roy being a sort of “embedded video-journalist” in our project, popping up every now and then to gather footage of what our team was doing. Though him not being based in Hong Kong or Addis Ababa wasn't ideal – he didn't visit that often, and predictably enough, it seemed that the most interesting stuff happened when he wasn't around.

Production Progresses, and the Concept Drifts

As Roy's film project progressed, I noticed in our intermittent conversations that his vision of the movie was drifting a bit from what we had originally discussed. I had wanted him to focus on the progress of the technology, and on the international team making the progress happen. But he was gravitating more toward focusing on ME as a person – on making it more of a biographical film. This didn't interest me nearly so much, mostly because I was more interested to tell the world about our AGI work than about myself. 

But in any case, I liked Roy and was happy to support his project even as his vision drifted from what we'd originally discussed. He started asking me for access to various people in my family and from earlier parts of my career, and I sent emails putting him in touch. He ended up interviewing a sort of quasi-random sample of people from my past and present, including my mom and dad and kids, and my first wife (but not my second), and a couple of my AI collaborators from the late 1990s. There were also quite a few people I told him it was important for him to interview – if he was going to do the “dig into Ben's past life and work” thing – but he didn't, probably mostly for cost reasons (e.g. he ended up interviewing a bunch of people who lived near New York, and omitting folks who lived in other areas).

As 2015 advanced Roy seemed very eager to finish the film and get it launched. I tried to convince him to take a slightly slower pace, and keep filming through 2016. It seemed to me that in 2016 we were likely to get OpenCog to control David Hanson's robots in a really interesting way, and that this would make a great ending for his film. But he didn't want to wait – I suppose understandably, because after all his funds were limited, and the timing of research is hard to predict. What if he waited through 2016 and then the OpenCog-controlled Hanson robot got deferred till 2017?

Choices, choices, choices...

I didn't see any rough cuts of the film while it was in progress (except one very crude, early trailer) and I had a lot of other things going on in my life, so I didn't think about it often. Then in early 2016 I saw that the film was to be shown at the Sheffield Documentary Festival; and Roy sent me a DVD.

I liked a lot of what I saw. On the other hand, out of all the footage he'd gathered, many of the choices he'd made were not what I had expected.

Overall, I felt upon first viewing, his film portrayed me as a charismatic, utopian, somewhat don-Quixote-ish character, tirelessly pursuing a wild-eyed dream of benevolent superhuman AI, persistently and enthusiastically ignoring the world's repeated pushbacks. 

The Ben in the film keeps moving to some new location, launching a new AI project and not quite getting to the finish line, then moving somewhere new and trying again: New York, Hong Kong, Ethiopia....  He makes a few personal and business messes due to caring more about his AI dream than anything else. But ever optimistic, enthusiastic and visionary, he keeps on pushing. And finally, by the end of the movie, he has found a powerful, equally starry-eyed and brilliant collaborator in roboticist David Hanson. Together, they will keep on pushing – and maybe they'll even get there eventually!

This Ben-character in Machine of Human Dreams certainly has a lot truth about him....  And it's certainly understandable that, to make an hour-long movie about a 49 year old person's life and work, a lot of simplifications and short-cuts will be needed. Nevertheless, given that the guy in the film is a bunch of samples of ME, I couldn't help, when I viewed the film, reflecting on everything that was left out, and the patterns of inclusions and omissions.

Overall, I can't really judge the quality of Machine of Human Dreams as a film in any objective way, I'm obviously too close to it. What I do feel impelled to do, though – and will do in the rest of this post – is step through the key episodes that the movie covers, and explain briefly what elements and aspects of the real-life versions of these episodes the film leaves out (for reasons of time limitation and choice of emphasis).

The film is not entirely chronological, but in my remarks here I'm going to proceed mostly in chronological order. The film starts with a sort of wild ride through my current work in Hong Kong and various interviews with people who are working with me here. This part is evocative, and shows some intriguing stuff. Then after that the film gets semi-chronological and quasi-autobiographical, beginning with my childhood.

AGI as a Crazy Hippie Dream

I gave Roy access to my mom for the movie, and he used a nice chunk of the interview footage he gathered with her. Watching this part was rather moving for me; my mom is a truly good-hearted, sincere and compassionate human being, as well as extremely competent and intelligent and hard-working. Hearing her recount bits and pieces about my early childhood on film was cool! Indeed it was my mom, in my first 4 years before I started school, who got me started on science and math and philosophy and creative imaginative thinking generally.

One key omission in the film pops up here: my other parent. While the interview footage Roy gathered from my father Ted Goertzel didn't make it into the film, Ted was also extremely important to me in my formative years. Ted was a sociology prof at U. of Oregon and then (for about 40 years, until he recently retired) at Rutgers University, and it was he who got me started on critical and analytical thinking. Ted has also followed my research career quite closely, including co-authoring and co-editing with me various books and papers on the future and social implications of AGI.

After my birth and infancy in Brazil, I lived from age 1.5 to ago 7 in Eugene Oregon. This was the late 1960s and early 1970s and the place was rather wild and full of hippies at the time. Ever since I have considered myself some sort of quasi-hippie – though obviously I'm too geeky, too hard-working and have too much of a hard-edged punk-rock/New-Yorker aesthetic to really be a hippie in the classic sense. The part of Machine of Human Dreams dealing with my childhood makes much of the roots of my AGI aspirations in the dreamy-eyed utopian idealism of the 60s/70s era. This is fair enough. Changing the world was what the adults around me were all about in Eugene back then; and I absorbed from my parents and their friends the idea that utterly changing the world was a reasonable thing to do and probably the most valuable way to spend one's life. Until my mid-teens I was fairly optimistic about radically improving the world via education and social reform; but at a certain point I shifted my views and became convinced that extreme technological advance was the best path toward tremendous positive transformation.

The film symbolizes the 60s/70s era culture by showing a bunch of freaks banging on drums in a field somewhere or other. I don't remember ever seeing anything quite like that. I do remember a near-constant stream of funky bearded guys in torn jeans and long-haired women in loose dresses and beads, singing folk music and strumming guitars and smoking weed … and lots and lots of political demonstrations, holding up signs and chanting and so forth. That was a time when folks in the counterculture believed anything was possible. I still feel that way.


Thinking About Thinking Machines

After my childhood, the movie briefly treats my undergraduate career, interviewing two people who knew me in college: my girlfriend (later first wife) Gwen, and my old friend Ken Silverman (who later worked with me at Webmind).

Ken recounts how, back when we were 15-17 years old and in our first couple years at Simon's Rock Early College, we used to sit around for hours and scheme about creating thinking machines. Well yes we did. Many of the theory-of-mind ideas I later published in “The Hidden Pattern” were already in my head back then. And Ken had lots of his own interesting ideas too, though he always tended to come back to hardware-focused solutions to AI problems, whereas I was focused more on the philosophical or mathematical aspects.

An understandable omission here is: The other good friends I also mused about AI with back then. For example: my college friend Mike Duncan who is still a close friend and who, unlike Ken, is still collaborating with me on AI at this moment, working on applications of OpenCog to analyzing genomics data, and on designing an OpenCog-friendly motivational and emotional system for David Hanson's robots. 

Ken's view is valid and interesting, but Mike's view would have been interesting to show too, especially because I haven't worked with Ken since 2001, whereas Mike has seen the details of my AI thinking and practical work all the way from 1984 through 2016. But Roy didn't happen to interview Mike just because the logistics didn't work out – while Ken lives in New York, Mike lives in Florida and Roy didn't have budget to haul a crew down there just to talk to him.

The movie skips everything I did between 1985 and 1997 – i.e. my grad school at NYU and Temple University, and then my whole academic career, in which I was a professor in departments of mathematics, computer science and psychology in the US, New Zealand and Australia. This was the time period in which I turned my vague college inklings about AI and cognitive science into more fleshed-out conceptual/mathematical theories (though still with many ambiguities and not nearly enough details to guide software implementation in detail). I also made some valuable collaborators in this period, e.g. Dr. Matt Ikle' whom I met in 1993 when we were colleagues in the math department at the University of Nevada Las Vegas, and who later became a co-creator of OpenCog's probabilistic logic and attention allocation subsystems.

The Webmind Experience – and my path toward practicality

The film gives a fair bit of airtime to Webmind Inc., a company I co-founded with 4 others in 1997 and that grew to around 130 total employees before it crashed and burned (alongside a lot of other cool dot-com era companies) in early 2001. It features a fair bit of Lisa Pazer, who was a Webmind co-founder and Webmind's first CEO. Her family also invested some seed money in Webmind in the very early stages; and as the film alludes, we were briefly romantically involved prior to co-founding Webmind (though that aspect of our relationship never went that far and ended before the company was started).

Right after Webmind tanked in 2001, I wrote an essay called “Waking Up from the Economy of Dreams” about the experience. While that piece of writing feels naïve and off-target to me now in some ways, it does effectively summarize my state of mind and attitude on the topic at the time. In hindsight, knowing what I do now about practical software projects and the tech business, all of us in the Webmind leadership – including me, Lisa, Ken Silverman, our second CEO Andy Siciliano, my long-time AI collaborator Cassio Pennachin (who first worked with me at Webmind, and is still working with me today) – were incredibly naïve. Ken and Cassio and I were naïve about how to manage and plan a project of the complexity of the Webmind AI engine; Lisa and Andy were naïve about how to run a technology R&D company of this complexity … we were all smart and ambitious and sincerely trying to do great things, but none of us really knew what we were doing in the context in which we were trying to operate.

Certainly, as Lisa alludes in her interview in the film, I was quite unrealistic at that stage of my life in terms of project planning – I was way overoptimistic in terms of how much work it takes to turn conceptual/mathematical designs into working large-scale software systems. And the business side of the company was not blessed with any particular skill at realism either. Nevertheless, at certain points the company did come pretty close to a successful exit via acquisition. There are some not-that-different parallel universes in which we sold that company at the right time, and made ourselves wealthy and came out of the experience looking like business wizards.

After Webmind shut its doors, I turned largely back to theory, and started thinking hard about how to incorporate everything I'd learned from the 3 years of science and engineering we'd done at Webmind, in a new AGI software design. My main goal was to encompass all the key ideas and structures in the Webmind design in an alternative design that would be much smaller and simpler. Webmind had been a wild grab-bag of different AI algorithms, all acting on the same “weighted labeled hypergraph” knowledge store. My new objective was to reduce the set of AI algorithms to a much smaller set, and to engineer these algorithms so that they would work very closely together. I still thought one needed an integrative, multi-algorithm approach to capture the richness and diversity of human intelligence, but I realized one had to be less willy-nilly about it, and carefully sculpt a set of algorithms intended to help each other out of their ruts. Eventually I came to call this principle “cognitive synergy” and I formulated it in a mathematical way.

Out of this phase of theory work came the AGI design I called the “Novamente Cognition Engine”, which eventually (in 2008) got open-sourced and morphed into OpenCog, and into the AGI design described in my 2014 books Engineering General Intelligence (co-authored with Cassio Pennachin and Nil Geisweiller).

One major part of my professional life that the film omits is my career in narrow-AI consulting and application development. In parallel with trying to work out a better thinking machine design, in the period 2001-2011 I also worked on a wide variety of practical AI consulting projects. I was based in Washington DC most of this time, and worked on bioinformatics for the NIH and CDC, and also (indirectly via various other entities) for INSCOM (Army intelligence), NSA and the Air Force. Some of the military/intelligence oriented work was interesting and potentially important, e.g. we used some tools from OpenCog to create software predicting which Army staff are most likely to commit suicide.

My bioinformatics consulting work over the years has largely been tied in with another, perhaps more critical aspect of my life that the film passes over -- my work on the application of AI to biology, and in particular to understanding the genomics of longevity. Alongside questing to build thinking machine, I have also spent a fair number of mind-cycles thinking about how to use AI technology to help cure aging and radically prolong human life. Some of my investigations in this area have been fairly successful, including new discoveries into the genetic roots of Chronic Fatigue Syndrome, Alzheimers and Parkinsons. I also helped Genescient Inc. understand why their super-long-lived flies live so long, and design some valuable nutraceuticals based on my AI analytics results.

This applied-AI aspect of my career is not that sexy or exciting for the most part (though, OK, the use of AI-driven bioinformatics to push toward a cure of aging and a path to superlongevity is arguably not all THAT boring!). On the other hand, the film generally gives the impression that I keep struggling and failing at everything in spite of my big ambitions and vision and immense knowledge, etc. It is certainly true that I have failed to create human-level AGI so far. But in my consulting work I have succeeded at some simpler (but not that simple) things, which have in some cases been highly rewarding and useful in themselves.

At the start of Webmind I was writing a lot of software code, but gradually as my career progressed I drifted into a pattern of coding only occasionally, and doing more theoretical and management work. The core of OpenCog, back when it was the Novamente Cognition Engine, was originally written by Andre' Senna and Thiago Maia in Brazil, working closely with Cassio Pennachin. Since the creation of OpenCog in 2008, the two most important contributors have been Dr. Linas Vepstas (based in Austin) and Dr. Nil Geisweiller (based in France and Bulgaria). Those guys have worked wonders. Nil is probably the only guy on the planet to fully understand my AGI design on the philosophical and mathematical level, and ALSO know the OpenCog codebase very well on a software level. Linas brings tremendous practical experience and software chops as well as deep mathematical and AI insight. Roy interviewed both of them for the film but ended up not using the footage – understandable perhaps given the time constraints of the movie, but still a bit distressing to me since these two extraordinarily brilliant and dedicated guys are really the ones most responsible for making the actual OpenCog system work.


My Hong Kong AI Adventures

In 2011 I relocated from DC to Hong Kong. I had been sick of DC on the personal level for a long time, but had been “stuck” there due to having a (roughly) 50-50 custody sharing arrangement with my ex-wife, Gwen, for our 3 kids. But by 2011 our youngest, Scheherazade, was about to do a junior year of high school overseas, so it seemed the right time to shift somewhere more interesting.

Machine of Human Dreams interviews Gwen a fair bit, and even has her appearing to say (via splicing together of utterances she made in different contexts while being interviewed) that she filed for divorce from me because I was so obsessed with AGI and my work that I couldn't pay enough attention to other people. Well, OK, whatever – I mean, the actual story of our divorce was definitely a lot more complex and nasty than that and didn't have much to do with AGI, but whatever. In this particular case the simplifications of the movie are probably in everybody's best interest….

In terms of omissions, it felt a bit odd to me on a personal level that he included Gwen in the movie but barely mentioned our 3 kids, e.g. our oldest son Zarathustra who is now getting his MS in computer science and moving toward a career in AI himself. But even more so, it felt odd that my long-ago first wife Gwen was included but he omitted my second wife Izabela Lyon Freire and – yeesh! – my actual wife Ruiting, to whom I've been happily married for 6 years now.  Both Ruiting and Izabela are excellent AI researchers with significant contributions to OpenCog. Izabela helped design OpenCog's PLN probabilistic reasoning engine.  Ruiting helped create OpenCog's natural language processing and dialogue subsystems.

Overall, through the years, my AGI obsession has drawn me to brilliant, active-minded women who are fascinated by AGI and other scientific and intellectual topics -- though it's true that Gwen's interest in these topics diminished as our marriage went on, and she got more focused on religion and nutrition, which interest me less.  And whatever difficulties I've had with love relationships have had little directly to do with my AGI obsession and more to do with other personality factors.

Roy skipped interviewing Izabela for the film because he didn't want to travel to Brazil; Gwen, being in the DC area, was more convenient. He did interview Ruiting fairly extensively, but in the final film she just shows up for a few seconds, jokingly noting that she doesn't want to move to Ethiopia (though the truth is, while she prefers living the developed world, she is willing to relocate with me to Ethiopia for a while if that turns out to be the best thing for our AGI work).

As an aside, Raj Dye's film Singularity or Bust has some sweet footage of Ruiting and me interacting with a Nao robot, back before we got romantically involved --you can clearly see the early sparks of our relationship there, which is pretty cool from my point of view...

The budding love relationship with Ruiting – who lived in Xiamen at the time – was part of the reason I relocated to Hong Kong. Another part was that Cassio and I secured funding to start a machine learning based investment management company, Aidyia Limited – which finally started trading a small fund in early 2016. And finally, Gino Yu and I got some Hong Kong government research funding for an OpenCog project at Hong Kong Poly U, where Gino is a professor.

The film omits mention of Aidyia, but a long middle segment of the film focuses on some robotics prototyping work we were doing at the Poly U OpenCog Lab in 2014-2015. This part of the film captures some cool robot-lab work and social dynamics, yet also feels to me like one of the more oddly focused segments. While Roy was at Poly U filming, the OpenCog team there was preparing to show some small robots to some officials from the grant funding agency that was funding the project, and there was some nervousness about putting on a good show. Roy liked this energy and nervousness, and for a while he was considering to focus his whole film mainly on our push to make a good demonstration for the funding agency officials.

I argued with Roy long and hard about this, at the time – making the points that: a) this demonstration had no particularly profound meaning, as it pertained to a robotics prototyping project that was valuable but not really core to the OpenCog AGI initiative; b) the people involved in the robotics prototyping work at Poly U were not really the key players in the OpenCog project anyway. Eventually I did convince him that focusing on this funding agency demo would result in a very boring film – I dragged him (metaphorically) kicking and screaming to Hanson Robotics and convinced him to end the film with David's gorgeous robots.

Still, though, I seem not to have fully convinced him – because that funding agency demo still absorbs a chunk of the film that is quite disproportionate to its actual importance. Also, the film omits the outcome of the demo, probably because it was boring … in the end, the demo we gave to the officials was underwhelming but adequate, and after a bureacratic delay of a couple months we were given a passing grade and the research (and research funding) continued.

The film then shows a few staff leaving the project after the underwhelming demo, hinting not too subtly at a potential causal connection between that demo and any staff departures; but in fact no such causal connection existed. Staff turnover was very high on our Hong Kong Poly U OpenCog project, mostly because the grant funding we had didn't allow us to pay market salaries. And the researcher who the film shows quitting OpenCog and leaving Hong Kong, Aaron Nitzkin, is a great guy and a deep cognitive theorist, but actually contributed fairly little to OpenCog due to his lack of professional programming skills.

This brings us to a big omission in this section of the film -- its failure to note the deepest OpenCog work was being done all through that time period by Linas and Nil in the US and Europe, far away from and almost completely ignorant of the Hong Kong team's robot prototype demos.

Also, as a minor point, the film shows OpenCog software and robotics developer Mandeep Bhatia musing about potentially moving back to India; but while Mandeep likes to think about this periodically, in fact he's still here in Hong Kong, now working with me and the rest of the team on making OpenCog control Hanson robots.

One thing the film does depict very accurately is that I have had a fascinating and fantastic time doing AI and robotics development here in Hong Kong. With David Hanson's amazing robot heads, Mark Tilden's walking robot bodies (briefly discussed at the start of the film) and OpenCog intelligence, we have the potential to make the smart, emotional, physically able humanoid robots everyone expects from science fiction. Ultimately AGI will transcend the human form and the human mind. But as we walk along that path, robots with humanoid form will have an important role to play in shaping the emergence of AGI cognition, emotion and values, and in helping human society come to grips with the onset of ever more advanced AGI. Despite some peculiarities of focus, I think the Hong Kong footage in Machine of Human Dreams does get across some of these themes in a striking and visual way.


iCog Labs and the Ethiopian Singularity

The film briefly shows me in Ethiopia discussing AI at a university there, and demo-ing robots out in the street with a team of young Ethiopian programmers. The robots in the streets of Addis look splendidly incongruous; and Ethiopian writer and tech project manager Hruy Tsegaye gives a beautiful speech about the power of advanced tech to advance Africa.

The vague impression given in the movie is that, after things got tough in Hong Kong, I started roaming far and wide in search of somewhere new to go and push forward with AGI – and I was so adventurous and maybe desperate that I looked as far as wildest Africa! This is indeed poetically true, in that I am very interested in the notion of building a large AGI team in Ethiopia where the costs are low, the people are lovely and the food is delicions. On the other hand, the dynamics of my involvement with Ethiopia has been a bit different than the film suggests.

I co-founded iCog Labs – Ethiopia's first AI/robotics firm -- in 2013 together with Ethiopian roboticist Getnet Aseffa Gezaw and American investor Sander Olsen. The idea for iCog originated when I visited Getnet in Addis Ababa in 2012, after getting to know him via the Internet in 2011. OpenCog and Hanson Robotics have been outsourcing work to iCog since 2013, and iCog has also been helping me with various AI consulting projects. I have consistently been impressed with the intelligence and ambition of the young computer scientists and programmers of Ethiopia. I have a few times considered relocating to Addis to work full-time on growing iCog into the world's greatest AGI, robotics and bioinformatics research center -- but at the moment I have a lot of interesting stuff going on here in Hong Kong, so I'm just visiting iCog as often as I can find room for.

Dr. Hanson's Robo-Dreams

One of my bigger successes here in Hong Kong has been to facilitate my good friend Dr. David Hanson moving his company here! David was coming to Hong Kong periodically before I moved here, because he was getting some robots made across the border in Shenzhen. On one of his visits, I introduced him to some of my tech-scene contacts here, and these contacts ended up garnering him investment money for his company Hanson Robotics – which ultimately ended up in him and his family moving out here to start a new branch of the firm, Hanson Robotics Hong Kong.

Working with David has been an intriguing, exhilarating, and sometimes exhausting experience. While his focus is mainly on emotional and social robotics, he fully gets my AGI vision and my intended route toward it with OpenCog. He has been both a good friend and an able collaborator … the end bit of Machine of Human Dreams quite accurately depicts what I'm doing with Hanson Robotics these days (well, as of 2015 anyway). The footage Roy got of the first version of David's “Sophia” robot is strikingly beautiful. The new version of Sophia is even better.

Since the time Roy wrapped up shooting Machine of Human Dreams we already have made great progress connecting OpenCog to the Hanson robots, and at time of writing, it seems it should be a small integer number of months before we have the first fully OpenCog-controlled Hanson robot head. Over the next few years, I think the Hanson robots can be both an outstanding showcase for OpenCog AGI, and a practically valuable medium for supplying OpenCog systems with the perception, action and social/emotional interaction they need to learn and grow.


Intelligent Networks Spawning Intelligent Networks ...

One thing that my work with David Hanson and his team – with their background in art and theater – has given me a stronger sense for, is the powerful urge the human mind has to perceive and create narrative structures. At some deep level, our hearts, minds and brains really want to view things in “Hero's Journey” type terms – in terms of stories with a beginning, middle and end … in terms of individual protagonists meeting obstacles and overcoming them and growing in the process, and so forth. We often get ourselves into trouble by unconsciously imposing this structure in cases where it doesn't really exist, or plays at most a minor role.

The great biologist Michael Rose, whom I worked with at Genescient Corp. for a few years (mostly at a distance, but occasionally face to face), often railed against this tendency as it manifested itself in biologists. Biological systems, as he understands them, are highly complex networks with subtle nonlinear self-organizing dynamics. Most meaningful biological effects emerge from rich networks of causation spanning numerous biological systems on multiple levels – many genes and proteins, many kinds of molecules, many kinds of cells, many organs.... Many biologists want to explain a disease or some other phenomenon via finding a single gene or a single biological process that is The Answer, or a single dynamic with a beginning, middle and end. But in reality, Michael emphasizes, biology doesn't work that way. There is no narrative. There are just mind-numbingly complex networks, out of whose distributed multilevel dynamics complex effects emerge.

I think Rose's complaint is also at the root of my complex, perplexed feelings toward the many simplifications made in Machine of Human Dreams. I have some difficulties with the tendency to simplify things into a templated, stereotyped narrative structure, even when this structure captures only a small part of the actual dynamics one cares about.

In its quest for clearly comprehensible drama and narrative simplification, one important thing the film de-emphasizes is that I've been supported in my passionate transcontinental quest for AGI by a rich and diverse network of friends, family and collaborators. The film sidesteps this aspect by focusing mostly on people from various stages of my life with whom I worked temporarily and stopped, and bypassing the other people from the same stages of my life with whom I've had strong, ongoing relationships. In this way, the film makes it look like I've been far more of a wandering loner than has been the case, and plays down the self-organizing social graph that has helped hugely in propelling my work forward. This makes ME seem like a more of a lone wolf and less of a human-network-aggregator than I actually am; and it makes the quest for building AGI seem like much less of a team effort than it really is.

What I'm doing with my life is not pushing to build a thinking machine all on my own – what I'm doing is serving as the seed about which a network of other brilliant people can crystallize, and providing a core of ideas to guide their work. This may seem like a fine distinction, but it's actually a very important one.

I remember one moment in my apartment in Hong Kong, when Roy was there filming along with a colleague from Roast Beef (whose name I've forgotten). His colleague said to me, while interviewing me, something like “You've moved around a lot. When you run into a dead end somewhere, you just cut your ties and move on, huh?” --- I looked at him bemused. What I said is something like, “No, not at all. When I get frustrated and want more opportunities, I do tend to move on to new places – but I never cut my ties. I've kept so many of the friends and colleagues I've had over the years all over the world. I'm actually really good at keeping touch with old friends and colleagues via the wonders of the Internet. And if you look at my colleagues now, there are people I've been working with since the 80s and 90s from all over the globe, flying in here to Hong Kong to collaborate. A bunch have even gone to Ethiopia with me to work with the guys there. The network of people collaborating on this stuff is not tied to any one physical location. However, I've often found that funding sources are obsessed with you being in a certain physical location....” – And then Roy's colleague quickly changed the subject, and the interview ended shortly after. My answer wasn't what he wanted to hear. He had already put me in the box of “Mr. Cuts His Ties and Moves On” because he was thinking of the narrative of the film that way. I got the feeling that -- unlike Roy -- this colleague wasn't really concerned about the actual human being or the actual science and engineering project that the film was supposed to be reporting on – he was more interested in coaxing me to say stuff that would fit into the narrative structure in his own head.

Similarly, as Ruiting recalls, when she was being interviewed for the film, they probed her with questions oriented toward finding controversy ...  like “Does Ben care more about his AGI work than he does about you? Does he prioritize his work over your relationship?” She answered that one something like “Sometimes, maybe” – which was both honest and sufficiently boring that it didn't make it into the film. But if they'd chosen to poke in different directions, they would have found Ruiting had a lot of interesting things to say on other relevant topics. 

For instance, when she started working on OpenCog-based natural language dialogue, she thought it would be a relatively easy problem, because OpenCog had a semi-magic logical inference engine that would just solve everything. Indeed, if your inference engine is good enough, then you can just pose every aspect of natural language dialogue as an inference problem, and you're done! But over years of thinking about it and working on it, she came to grips more thoroughly with the nature of the problem – which is that, in order to perform usefully fast on linguistic problems, the inference engine needs to be guided by linguistic knowledge … but the linguistic knowledge can only be gathered by inference … so you actually have a “chicken or egg” problem … you have a complex cognitive system in which each part requires the others in order to function. 

How Ruiting's thinking about AGI and language processing has matured over the 6 years we've worked together would be a challenge to portray in a film – however, even a slight hint in this direction would have been interesting to show I think. As it is, the film doesn't even hint that this sort of aspect exists in my life – the fact that I'm joyfully married to a lovely young woman who is working and thinking together with me about AGI is utterly bypassed.

This hits on a larger point: It occurs to me that, if one looks at Machine of Human Dreams as a portrayal of my own personal journey through AGI development from a classic-narrative “Hero's Quest” type perspective, one major thing missing is any substantial depiction of the growth and transformation the “hero” goes through as a result of his trials and tribulations. The film gets across that I'm a dude who holds AGI as a Grand Goal and keeps on trying, and whose frustrations never last too long – when I hit an obstacle or setback I do sometimes get pissed off or even temporarily depressed, but ultimately I just ram into it again or look for some other way around, because I can see so damn clearly in my mind what's on the other side! But the film just shows me keeping on going, and doesn't really show how I've grown and adapted as result of keeping on going for 49 years. Compared to the real Ben Goertzel, the Ben G character in Machine of Human Dreams is a lot more Energizer Bunny like – he keeps on going and going admirably, but he never really changes.

In fact I've changed a lot in my life, in various phases, in multiple ways that are relevant to my AGI work. In the Webmind era I was fantastically unrealistic in my project planning and time estimates. I'm still a bit on the optimistic side, to be sure; but I'm now more within the scope of ordinary optimistic project leaders – there's a world of difference. I have undertaken great efforts to rid my mind of delusions insofar as possible, to really see what is feasible versus what I'd like to be feasible, to clearly distinguish intuition from solid knowledge and research projects from engineering projects. Again I have not become a hard-nosed pragmatist but I've become way way better at distinguishing how the world is from how I'd like it to be – while still pushing to make it become more the way I'd like it to be! Much of this learning has occurred as a result of doing various practical, applied narrow-AI consulting projects, a side of my life that is far more boring than the quest for superhuman AGI ... but yet if I do succeed at building superhuman AGI, this will partly be due to the modicum of discipline I learned from spending a bunch of time delivering real stuff for customers, sometimes successfully and sometimes not.

On a more personal side, I've gradually had to learn to stress out less about the frustration of my grand cosmic goals being so slow to achieve; and of spending so much of my time on stuff that I enjoy only moderately and that works only indirectly toward my grand goals (e.g. managing people, doing consulting projects, going around seeking funding,...). At times this has really made me feel like shit; but eventually, through various sorts of efforts and relaxations, I've untied a lot of knots in my mind and become OK with everything. These days I feel a deep inner contentment, even while working like hell toward difficult goals in chaotic situations – a feeling that visited me only much more intermittently in previous parts of my life. Most likely some of the crappy decisions I made in earlier phases of my life were rooted in a deep inner discontent, which plays a much lesser role in my psyche these days.

The film's portrayal of me as an obsessed AGI zealot is certainly accurate; but my own individual growth and transformations, which have impacted the flow of my AGI work in huge ways, have been driven as much by my various non-AGI passions and occupations – for instance, my explorations with psychedelics, and my research into psi phenomena, both of which are left out of Machine of Human Dreams for understandable reasons (all that is fascinating and important stuff, but would be distracting and hard to capture usefully in a brief way). As it happens, the three people who could have infused the film with some insight into the diverse factors driving my evolution as a human being and scientist – my wife Ruiting, my oldest son Zar, and my ex-wife Izabela – were omitted from the film, although Roy did interview Ruiting and Zar fairly extensively.

I can't especially fault Roy Cohen for not coming to grips with the depths of Ben Goertzel's psyche … I'm a complex, unique sort of weirdo, and Roy and I never had the kinds of conversations that would have enabled him to really understand me well. Roy and his colleagues always stayed pretty close to the surface in their interview questions; and I never tried to push them into grokking my individual character more thoroughly, mostly because I wanted their film to focus on OpenCog and the quest for superhuman AGI rather than me as a person. But of course, if you really want to dig deep deep deep into things, the motivations and networks inside my human mind and the motivations and networks in the AGI and OpenCog communities and the motivations and networks inside the still-incipient OpenCog AGI minds are all dynamically interpenetrating and growing in a coupled way. This shit is bloody complicated!

Conveying the complex social-dynamical phenomena via which networks of intelligent people are coming together to create networks of intelligent processing inside AGI systems … together with the inner growth and transformations of the people trying to crystallize such networks around breakthrough ideas – sure, this would be a big challenge from a film-making perspective, and I can see why Roy and his colleagues found it convenient to fall back on more of a standard narrative structure....

Net net, while it omits various relevant aspects of the underlying reality, still, the story Roy tells is pretty good. I do feel there's an even more fascinating kind of story lurking beneath. But I suppose that's usually the case...

Meanwhile, a film by nature is frozen in time, whereas reality moves on. Just in the last week -- i.e. 8 months or so after Roy finished shooting Machine of Human Dreams -- we've gotten OpenCog fully hooked up to the gorgeous Hanson robots, controlling their verbal and nonverbal behavior. Now it's down to making the robot smarter and smarter.

We're doing a low-cost-robot soccer contest in Ethiopia later this year and my hope is to have one of the Hanson robots give the speech at the opening ceremony, maybe with a funky African-style braided wig. Lots of high Ethiopian government folks will be there, I'm sure their jaws will drop. And we're still jamming with Mark Tilden, moving forward on plans for a humanoid walking body to go along with the Hanson heads and OpenCog mind.

And behind the scenes the AI keeps progressing. Nil (French, living in Bulgaria) and Eddie (from Vermont, but was just here in Hong Kong for a couple weeks) and Misgana (moved from the Ethiopia office to Hong Kong some time ago) have gotten OpenCog's probabilistic inference engine (first described in a 2006 book by me, my old Las Vegas + New York collaborator Matt Ikle', my ex-wife Izabela and my Finnish transhumanist friend Ari Heljakka) to do some cool inferences about the biology of longevity, as well as about what people say to the robot. The non-linear-dynamical attention allocation math that Matt and I worked out over a decade ago is actually finally working now, thanks to some work by Misgana and with some help from Roman, a German intern who also wrote a Lojban interface to OpenCog (Lojban is a speakable form of predicate logic, around for more than half a century spoken by a small community of awesome geeks on the Internet).

Overall – the international network of human minds is gradually bringing to life a plausible approximation of the cognitive network that Nil, Cassio and I described in Engineering General Intelligence in 2014. This emerging mind network is starting to display itself via the emoting faces of David Hanson's beautiful robots but also in other ways, such as finding patterns in complex networks of genomic data.

A new form of life is unfolding, little by little. Very fast on the historical scale, sometimes painstakingly slowly on the time-scale of daily endeavor. Machine of Human Dreams depicts a few interesting fragments of the process, captured during a brief slice of time. The AI work will progress a bunch further inbetween me writing these words and you reading them. 

Us personalities involved in building AGI and our individual narratives and stories are often colorful and interesting, but from a bigger view, we're kinda like the funky flashes of fire coming out of the bottom of a rocket as it blasts into space. Yes, the dancing yellow flames from the rocket are fascinating, and you can stare at them a while and get lost. But the flames on a campfire are fascinating also. The unique thing about the rocket is that it's going into space. The unique thing about the story Roy Cohen captured is that this is a group of people building a mind beyond the human. Even though I'm working toward this goal every day, and the step by step work can be difficult and tedious, it still blows my mind to think about what we're doing and its ultimate implications.










Sunday, March 20, 2016

Transparent AGI: Benefits and Risks


A recent article by Bill Hibbard, myself and other colleagues on the value of openness in AI development (linked to a petition in favor of transparent AI development and deployment) has, predictably, gotten a bunch of comments from people alarmed at the potential riskiness of putting AGI designs and code in the hands of the general public.

Another complaint from many commenters has been that there is so much profit in AGI that big companies will inevitably dominate it, buying any small actors who make significant innovations.

The second complaint has a more obvious counter-argument – simply that not everyone can be bought by big companies, either because

  • they are already rich due to non-AI stuff, like the folks behind OpenAI
  • they are “crazy / idealistic”, like those of us behind OpenCog

So if some folks who cannot be bought make enough progress to seed a transparent and open source AGI movement that takes off fast enough, it may beat the big tech companies. Linux provides an inexact but meaningful analogue; in some important ways it's beating the big companies at OS development. And many of the individuals behind Linux are too crazy/idealistic to be bought by big companies.

The counterargument to the first complaint is slightly subtler, but has some important ingredients in common with the “seed a movement” counterargument to the second complaint. Saying “it's dangerous to give AGI to bad guys” overlooks the complexity of the likely processes underlying the development of AGI, and the relation of these processes to the nature of the AGI itself and the social networks in which emerging AGIs are embedded.

In this blog post I will explore some of this complexity. This is somewhat of a preliminary document and these ideas will likely get presented in a better organized and more systematic fashion later, somewhere or other....

First: it is true that, given any specific AGI system, it MIGHT be safer to keep that particular AGI system in the hands of a specific chosen group of beneficial actors, than to spread knowledge of that AGI system widely. Whether this is actually true in a particular instance depends on various factors including:

  • how beneficial that chosen group actually is
  • the number and nature of “smart but malevolent or irresponsible” other parties on the scene
  • how much resources the AGI requires to run effectively
  • whether the rest of the world will get annoyed and start a fight about having the secrets of AGI kept from it (and then who is likely to win that fight)
  • etc.

However, looking at it this way overlooks many things, including the dependency between two factors:

  • the degree of openness with which an AGI system is developed
  • the various properties (e.g. robustness and general beneficial-ness) of that AGI system

One could argue that the RIGHT small, elite, closed group is more likely to develop a robust and beneficial AGI system than a large, distributed, diverse motley crew of participants (such as tends to emerge in a successful OSS project community). On the other hand, this argument seems very weak, because in an open source setting, there is still the opportunity for altruistic smart, elite group to fork existing codebases and create their own separate version, leveraging the work done by others but perfecting it according to their own ideas and aesthetics. One could argue that closed-source provides more incentives for smart, elite groups to participate in projects. But again this is weak, given the evidence of well-funded OSS AI initiatives like OpenAI, and given the high level of technical strength of many individuals in the broader OSS community. There are more big proprietary projects than big OSS projects out there. But the big OSS projects are generally very high in quality.

Commercial projects have historically done better at user interface refinement than OSS projects. But creating AGI is arguably more like building an OS kernel or a machine learning library than like building a user interface – i.e. it's complex and technically subtle, requiring deep expertise to make real progress. This is the kind of thing the OSS world has been good at. (Indeed, similar to user interfaces, we need AGIs to respond to the various peculiarities of human nature. But we need the AGI to learn this kind of response, we don't want to code the particulars of human nature into the AGI one by one. And this kind of learning appears to require algorithms that appear highly technical and tricky given the current state of science.)

My own feeling is that an open and transparent modality is much more likely to lead to a robust and beneficial AGI. This is because there will be such a diverse group of smart people working on it. And because the group of people working on it will not be biased by having specific commercial goals. Even when the business goals underlying a commercial AGI system are not especially nefarious nor contradictory to the goal of making broadly beneficial AGI – nevertheless, the existence of specific commercial goals will bias the thinking of the people involved in a certain direction, leading them to overlook certain promising directions and also certain risks.

As Bill Hibbard points out in his recent follow-up article, the “is open versus closed AGI better” debate ties in closely with differing ideas about AGI design and the likely process via which AGI will develop. If one believes there is going to be a relatively small/simple AGI design, which will give anyone who “knows the trick” dramatically superior performance to anything that came before it – then there is a reasonable argument for keeping this trick secret, assuming one trusts the group that holds the secret. If one believes that the first powerful AGI is likely to be more complex and heterogeneous, emergent from the combination of a large number of different software components carrying out different functions in different ways, then there is less argument for keeping such systems secret as they develop.

For one thing, in the latter “big emergent combination” scenario, secrets about AGI design will not likely be well kept anyway. Big tech companies are far outpacing top-secret underground government labs in AI development, and this trend seems likely to continue; but big companies have ongoing employee turnover and tend not to be extremely good at keeping secrets for long periods of time. If AGI is going to emerge via a pathway that requires years of effort and incremental improvements, then the in-process AGI system is bound to leak out even if it's developed in a big-company lab. (Whether a top-secret government lab would be any better at keeping a complex AGI design secret is a different question. I doubt it; there are plenty of spies and double agents about....)
For another thing, a complex, heterogeneous system is exactly the sort of thing that a large, diverse community has a lot to contribute to. Parts of such a system that are not especially critical to any company's business model, can nonetheless get loving care from some brilliant, focused academic or hacker community.

In principle, of course, if a company or government were rich enough and ambitious enough, they could buy an almost arbitrarily diverse development community. Taking this to the ultimate extent one has a fascist-type model where some company or government agency rigidly controls everyone in the world -- a Google-archic government, or whatever. But in practice any company or government agency seems to only be able to acquire relatively limited resources, not sufficient to enable them to fund top-notch teams working on peripheral aspects of their AI projects.

So given all the above, it seems we may well have a choice between

  • a worse (less robust, less richly and generally intelligent) AGI that is created and owned by some closed group
  • a better (more robust, more richly and generally intelligent) AGI that is created and deployed in an open way, and not owned by anyone

Given this kind of choice, the risk of a nasty group of actors doing something bad with an AGI would not be the decisive point. Rather, we need to look at multiple options such as

  1. the odds of a nasty group getting ahold of an AGI and things going awry
  2. the odds of a group with broadly beneficial goals in mind getting ahold of an AGI and things going awry
  3. the odds of a group with non-malevolent but relatively narrow (e.g. commercial or national-security) goals getting ahold of an AGI and things going awry

On the face of it, Problem 1 would seem more likely to occur with the open approach. But if open-ness tends to lead to more robust and beneficial AGIs (as I strongly suspect is the case) then Problems 2 and especially 3 are more likely to occur in the closed case.

One must bear in mind also that there are many ways for things to go awry. For instance, things can go awry due to what an AGI directly does when deployed in the world. Or, things can go awry due to the world's reaction to what an AGI does when deployed in the world. This is a point Bill Hibbard has highlighted in his response. A closed commercial AGI will more likely be viewed as manipulating people for a the good of a small elite group, and hence more likely to arouse public ire and cause potential issues (such as hackers messing with the AGI, for that matter). A closed military AGI has strong potential to lead to arms-race scenarios.

Summing up, then -- I don't claim to have a simple, knockout argument why transparent, open AGI is better. But I want to emphasize that the apparent simple, knockout argument why closed AGI is better, is simply delusory. Saying “closed AI is better because with open AI, bad guys can take it and kill us all” is simply sweeping copious complexities of actual reality under the rug. It's roughly analogous to saying “having more weapons is better because then it's possible to blow up a larger class of enemies.” The latter problematically overlooks arms-race phenomena, in which the number of weapons possessed by one group, affects the number of weapons possessed by another group; and also psychological phenomena via which both adversarial and beneficial attitudes tend to spread. The former problematically overlooks the dependencies between the open vs. closed choice and the nature of the AGI to be developed, and the nature of the social network in which the AGI gets released.

I have considered a bunch of complexly interdependent factors, in the above.  They're all quite clear in my own head, but I worry that due to typing these notes hastily, I may not have made it all that clear to the reader.   Some futurists have proposed a Bayesian decision-tree approach to describing complex future situations involving subtle choices. It might be interesting to make such a tree regarding closed vs. transparent AGI, based on the factors highlighted in this blog post along with any others that come up.   This might be a more effective way of clearly outlining all the relevant issues and their cross-relationships.

Monday, November 30, 2015

Getting Human-Like Values into Advanced OpenCog AGIs

Some Speculations Regarding Value Systems for Hypothetical Powerful OpenCog AGIs

In a recent blog post, I have proposed two general theses regarding the future value systems of human-level and transhuman AGI systems: the Value Learning Thesis (VLT) and Value Evolution Thesis (VET).   This post pursues the same train of thought further – attempting to make these ideas more concrete via speculating about how the VLT and VET might manifest themselves in the context of an advanced version of the OpenCog AGI platform.  

Currently OpenCog comprises a comprehensive design plus a partial implementation, and it cannot be known with certainty how functional a fully implemented version of the system will be.   The OpenCog project is ongoing and the system becomes more functional each year.  Independent of this, however, the design may be taken as representative of a certain class of AGI systems, and its conceptual properties explored.

An OpenCog system has a certain set of top-level goals, which initially are supplied by the human system programmers.   Much of its cognitive processing is centered on finding actions which, if executed, appear to have a high probability of achieving system goals.  The system carries out probabilistic reasoning aimed at estimating these probabilities.   Though from this view the goal of its reasoning is to infer propositions of the form “Context & Procedure ==> Goal”, in order to estimate the probabilities of such propositions, it needs to form and estimate probabilities for a host of other propositions – concrete ones involving its sensory observations and actions, and more abstract generalizations as well.   Since precise probabilistic reasoning based on the total set of the system’s observations is infeasible, numerous heuristics are used alongside exact probability-theoretic calculations.   Part of the system’s inferencing involves figuring out what subgoals may help it achieve its top-level goals in various contexts.

Exactly what set of top-level goals should be given to an OpenCog system aimed at advanced AGI is not yet fully clear and will largely be determined via experimentation with early-stage OpenCog systems, but a first approximation is as follows, determined via a combination of theoretical and pragmatic considerations.    The first four values on the list are drawn from the Cosmist ethical analysis presented in my books A Cosmist Manifesto and The Hidden Pattern; the others are included for fairly obvious pragmatic reasons to do with the nature of early-stage AGI development and social integration.  The order of the items on the list is arbitrary as given here; each OpenCog system would have a particular weighting for its top-level goals.

  • Joy: maximization of the amount of pleasure observed or estimated to be experienced by sentient beings across the universe
  • Growth: maximization of the amount of new pattern observed or estimated to be created throughout the universe
  • Choice: maximization of the degree to which sentient beings across the universe appear to be able to make choices (according e.g. to the notion of “natural autonomy”, a scientifically and rationally grounded analogue of the folk notion and subjective experience of “free will”)
  • Continuity:  persistence of patterns over time.   Obviously this is a counterbalance to Growth; the relative weighting of these two top-level goals will help determine the “conservatism” of a particular OpenCog system with the goal-set indicated here.
  • Novelty: the amount of new information in the system’s perceptions, actions and thoughts
  • Human pleasure and fulfillment: How much do humans, as a whole, appear to be pleased and fulfilled?
  • Human pleasure regarding the AGI system itself: How pleased do humans appear to be with the AGI system, and their interactions with it?
  • Self-preservation: a goal fulfilled if the system keeps itself “alive.”   This is actually somewhat subtle for a digital system.    It could be defined in a copying-friendly way, as preservation of the existence of sentiences whose mind-patterns have evolved from the mind-patterns of the current system this with a reasonable degree of continuity.

·      This list of goals has a certain arbitrariness to it, and no doubt will evolve as OpenCog systems are experimented with.   However, it comprises a reasonable “first stab” at a “roughly human-like” set of goal-content for an AGI system.

One might wonder how such goals would be specified for an AGI system.   Does one write source-code that attempts to embody some mathematical theory of continuity, pleasure, joy, etc.?    For some goals mathematical formulae may be appropriate, e.g. novelty which can be gauged information-theoretically in a plausible way.   In most cases, though, I suspect the best way to define a goal for an AGI system will be using natural human language.   Natural language is intrinsically ambiguous, but so are human values, and these ambiguities are closely coupled and intertwined.   Even where a mathematical formula is given, it might be best to use natural language for the top-level goal, and supply the mathematical formula as an initial suggest means of achieving the NL-specified goal.   
The AGI would need to be instructed – again, most likely in natural language – not to obsess on the specific wording supplied to it in its top-level goals, but rather to take the wording of its goals as indicative of general concepts that exist in human culture and can be expressed only approximatively in concise sequences of words.     The specification of top-level goal content is not intended to precisely direct the AGIs behavior in the way that, say, a thermostat is directed by the goal of keeping temperature within certain bounds.  Rather, it is intended to point the AGI’s self-organizing activity in certain informally-specified directions.

Alongside explicitly goal-oriented activity, OpenCog also includes “background processing” – cognition simply aimed at learning new knowledge, and forgetting relatively unimportant knowledge.   This knowledge provides background information useful for reasoning regarding goal-achievement, and also builds up a self-organizing, autonomously developing body of active information that may sometimes lead a system in unpredictable directions – for instance, to reinterpretation of its top-level goals.

The goals supplied to an OpenCog system by its programmers are best viewed as initial seeds around which the system forms its goals.  For instance, a top-level goal of “novelty” may be specified as a certain mathematical formula for calculating the novelty of the system’s recent observations, actions and thoughts.  However, this mathematical formula may be intractable in its most pure and general form, leading the system to develop various context-specific approximations to estimate the novelty experienced in different situations.   These approximations, rather than the top-level novelty formula, will be what the system actually works to achieve.   Improving these approximations will be part of the system’s activity, but how much attention to pay to improving these approximations will be a choice the system has to make as part of its thinking process.    Potentially, if the approximations are bad, they might cause the system to delude itself that it is experiencing novelty (according to its top-level equation) when it actually isn’t, and also tell the system that there is no additional novelty to be found in in improving its novelty estimation formulae.  

And this same sort of problem could occur with goals like “help cause people to be pleased and fulfilled.”   Subgoals of the top-level goal may be created via more or less crude approximations; and these subgoals may influence how much effort goes into improving the approximations.   Even if the system is wired to put a fixed amount of effort into improving its estimations regarding which subgoals should be pursued in pursuit of its top-level goals, the particular content of the subgoals will inevitably influence the particulars of how the system goes about improving these estimations.
The flexibility of an OpenCog system, its ability to ongoingly self-organize, learn and develop, brings the possibility that it could deviate from its in-built top-level goals in complex and unexpected ways.  But this same flexibility is what should – according to the design intention – allow an OpenCog system to effectively absorb the complexity of human values.   Via interacting with humans in rich ways – not just via getting reinforced on the goodness or badness of its actions (though such reinforcement will impact the system assuming it has goals such as “help cause human pleasure and fulfillment”), but via all sorts of joint activity with humans – the system will absorb the ins and outs of human psychology, culture and value.   It will learn subgoals that approximately imply its top-level goals, in a way that fits with human nature, and with the specific human culture and community it’s exposed to as it grows.

In the above I have been speaking as if an OpenCog system is ongoingly stuck with the top-level goals that its human programmers have provided it with; but this is not necessarily the case.   Operationally it is unproblematic to allow an OpenCog system to modify its top-level goals.   One might consider this undesirable, yet a reflection on the uncertainty and ignorance necessarily going into any choice of goal-set may make one think otherwise.  

A highly advanced intelligence, forced by design to retain top-level goals programmed by minds much more primitive than itself, could develop an undesirably contorted psychology, based on internally working around its fixed goal programming.   Examples of this sort of problem are replete in human psychology.  For instance, we humans are “programmed” with a great deal of highly-weighted goal content relevant to reproduction, sexuality and social status, but the more modern aspects of our minds have mixed feelings about these archaic evolved goals.   But it is very hard for us to simply excise these historical goals from our minds.   Instead we have created quite complex and subtle psychological and social patterns that indirectly and approximatively achieve the archaic goals encoded in our brains, while also letting us go in the directions in which our minds and cultures have self-organized during recent millennia.    Hello Kitty, romantic love, birth control, athletic competitions, investment banks – the list of human-culture phenomena apparently explicable in this way is almost endless.

One key point to understand, closely relevant to the VLT, is that the foundation of OpenCog’s dynamics in explicit probabilistic inference will necessarily cause it to diverge somewhat from human judgments.   As a probabilistically grounded system, OpenCog will naturally try to accurately estimate the probability of each abstraction it makes actually applying in each context it deems relevant.    Humans sometimes do this – otherwise they wouldn’t be able to survive in the wild, let alone carry out complex activities like engineering computers or AI systems – but they also behave quite differently at times.   Among other issues, humans are strongly prone to “wishful thinking” of various sorts.   If one were to model human reasoning using a logical formalism, one might end up needing to include a rule of the rough form

X would imply achievement of my goals
therefore
X’s truth value gets boosted

Of course, a human being who applied this rule strongly to all X in its mind, would become completely delusional and dysfunctional.  No human is like that.  But this sort of wishful thinking infuses human minds, alongside serious attempts at accurate probabilistic reasoning, plus various heuristics which have various well-documented systematic biases.   Belief revision combines conclusions drawn via wishful thinking, with conclusions drawn by attempts at accurate inference, in complex and mainly unconscious ways.  

Some of the biases of human cognition are sensible consequences of trying to carry out complex probabilistic reasoning on complex data using limited space and time resources.  Others are less “forgivable” and appear to exist in the human psyche for “historical reasons”, e.g. because they were adaptive for some predecessor of modern humanity in some contexts and then just stuck around.
An advanced OpenCog AGI system, if thoroughly embedded in human society and infused with human values, would likely arrive at its own variation of human values, differing from nearly any human being’s particular value system in its bias toward logical and probabilistic consistency.   The closest approximation to such an OpenCog system’s value system might be the values of a human belonging to the human culture in which the OpenCog system was embedded, and who also had made great efforts to remove any (conscious or unconscious) logical inconsistencies in his value system.

What does this speculative scenario have to say about the VLT and VET?  

Firstly, it seems to support a limited version of the VLT.   An OpenCog system, due to its fundamentally different cognitive architecture, is not likely to inherit the logical and probabilistic inconsistencies of any particular human being’s value system.  Rather, one would expect it to (implicitly and explicitly) seek to find the best approximation to the value system of its human friends and teachers, within the constraint of approximate probabilistic/logical consistency that is implicit in its architecture.  

The precise nature of such a value system cannot be entirely clear at this moment, but is certainly an interesting topic for speculative thinking.    First of all, it is fairly clear which sorts of properties of typical human value systems would not be inherited by an OpenCog of this hypothetical nature.   For instance, humans have a tendency to place a great deal of extra value on goods or ills that occur in their direct sensory experience, much beyond what would be justified by the increased confidence associated with direct experience as opposed to indirect experience.   Humans tend to value feeding a starving child sitting right in front of them, vastly more than feeding a starving child halfway across the world.  One would not expect an reasonably consistent human-like value system to display this property.

Similarly, humans tend to be much more concerned with goods or ills occurring to individuals who share more properties with themselves – and the choice of which properties to weight more highly in this sort of judgment is highly idiosyncratic and culture-specific.    If an OpenCog system doesn’t have a top-level goal of “preserving patterns similar to the ones detected in my own mind and body”, then it would not be expected to have the same “tribal” value-system bias that humans tend to have.    Some level of “tribal” value bias can be expected to emerge via abductive reasoning based on the goal of self-preservation (assuming this goal is included), but it seems qualitatively that humans have a much more tribally-oriented value system than could be derived via this sort of indirect factor alone.   Humans evolved partially via tribe-level group selection; an AGI need not do so, and this would be expected to lead to significant value-system differences.    

Overall, one might reasonably expect an OpenCog created with the above set of goals and methodology of embodiment and instruction to arrive at a value system that is roughly human-like, but without the glaring inconsistencies plaguing most practical human value systems.   Many of the contradictory aspects of human values have to do with conflict between modern human culture and “historical” values that modern humans have carried over from early human history (e.g. tribalism).   One may expect that, in the AGI’s value system, the modern culture side of such dichotomies will generally win out – because it is what is closer to the surface in observed human behavior and hence easier to detect and reason about, and also because it is more consilient with the explicitly Cosmist values (Joy, Growth, Choice) in the proposed first-pass AGI goal system.  

So to a first approximation, one might expect an OpenCog system of this nature to settle into a value system that
  • Resembles the human values of the individuals who have instructed and interacted with it
  • Displays a strong (but still just approximate) logical and probabilistic consistency and coherence
  • Generally resolves contradictions in human values via selecting modern-culture value aspects over “archaic” historical value aspects


It seems likely that such a value system would generally be acceptable to human participants in modern culture who value logic, science and reason (alongside other human values).    Obviously human beings who prefer the more archaic aspects of human values, and consider modern culture largely an ethical and aesthetic degeneration, would tend to be less happy with this sort of value system.  

So in this view, an advanced OpenCog system appropriately architected and educated would validate the VLT, but with a moderately loose interpretation.   Its value system would be in the broad scope of human-like value systems, but with a particular bias and with a kind of consistency and purity not likely present in any particular human being’s value system.

What about the VET?   It seems intuitively likely that the ongoing growth and development of an OpenCog system as described above would parallel the growth and development of human uploads, cyborgs or biologically-enhanced humans who were, in the early stage of their posthuman evolution, specifically concerned with reducing their reliance on archaic values and increasing their coherence and logical and probabilistic consistency.   Of course, this category might not include all posthumans – e.g. some religious humans, given the choice, might use advanced technology to modify their brains to cause themselves to become devout in their particular religion to a degree beyond all human limits.   But it would seem that an OpenCog system as described above would be likely to evolve toward superhumanity in roughly the same direction as a human being with transhumanist proclivities and a roughly Cosmist outlook.    If indeed this is the case, it would validate the VET, at least in this particular sort of situation.

It will certainly be noted that the value system of “a human being with transhumanist proclivities and a Cosmist outlook” is essentially the value system of the author of this article, and the author of the first-pass, roughly sketched OpenCog goal content used as the basis of the discussion here.   Indeed, the goal system outlined above is closely matched to my own values.   For instance, I tend toward technoprogressivism as opposed to transhumanist political libertarianism – and this is reflected in my inclusion of values related to the well-being of all sentient beings, and lack of focus on values regarding private property.   

In fact, different weightings of the goals in the above-given goal-set would be expected to lead to different varieties of human-level and superhuman AGI value system – some of which would be more “technoprogressivist” in nature and some more “political libertarian” in nature, among many other differences.   In a cosmic sense, though, this sort of difference is ultimately fairly minor.  These are all variations of modern human value system, and occupy a very small region in the space of all possible value systems that could be adopted by intelligences in our universe.   Differences between different varieties of human value system often feel very important to us now, but may well appear quite insignificant to our superintelligent descendants.