Amara Angelica pointed me to an article in IEEE Spectrum titled MoNETA: A Mind Made from Memristors
Fascinating indeed!
I'm often skeptical of hardware projects hyped as AI projects, but truth be told, I find this one an extremely exciting and promising project.
I think the memristor technology is amazing and may well play part in the coming AGI revolution.
Creating emulations of human brain microarchitecture is one fascinating application of memristors, though not the only one and not necessarily the most exciting one. Memristors can also be used to make a lot of other different AI architectures, not closely modeled after the human brain.
[For instance, one could implement a semantic network or an OpenCog-style AtomSpace (weighted labeled hypergraph) via memristors, where each node in the network has both memory and processor resident in it ... this is a massively parallel network implemented via memristors, but the nodes in the network aren't anything like neurons...]
And, though the memristors-for-AGI theme excites me, this other part of the article leaves me a bit more skeptical:
"
By the middle of next year, our researchers will be working with thousands of candidate animats at once, all with slight variations in their brain architectures. Playing intelligent designers, we'll cull the best ones from the bunch and keep tweaking them until they unquestionably master tasks like the water maze and other, progressively harder experiments. We'll watch each of these simulated animats interacting with its environment and evolving like a natural organism. We expect to eventually find the "cocktail" of brain areas and connections that achieves autonomous intelligent behavior.
"
I think the stated research program places too much emphasis on brain microarchitecture and not enough on higher-level cognitive architecture. The idea that a good cognitive architecture is going to be gotten to emerge via some simple artificial-life type experiments seems very naive to me. I suspect that, even with the power of memristors, designing a workable cognitive architecture is going to be a significant enterprise. And I also think that many existing cognitive architectures, like my own OpenCog or Stan Franklin's LIDA or Hawkins' or Arel's deep learning architectures, could be implemented on a memristor fabric without changing their underlying concepts or high-level algorithms or dataflow.
So: memristors for AI, yay!
But: memristors as enablers of a simplistic Alife approach to AGI ... well, I don't think so.
Tuesday, November 23, 2010
The Psi Debate Continues (Goertzel on Wagenmakers et al on Bem on precognition)
A few weeks ago I wrote an article for H+ Magazine about the exciting precognition results obtained by Daryl Bem at Cornell University.
Recently, some psi skeptics (Wagenmakers et al) have written a technical article disputing the validity of Bem's analyses of his data.
In this blog post I'll give my reaction to the Wagenmakers et al (WM from here on) paper.
It's a frustrating paper, because it makes some valid points -- yet it also confuses the matter by inappropriately accusing Bem of committing "fallacies" and by arguing that the authors' preconceptions against psi should be used to bias the data analysis.
The paper makes 3 key points, which I will quote in the form summarized here and then respond to one by one
POINT 1
"
Bem has published his own research methodology and encourages the formulation of hypotheses after data analysis. This form of post-hoc analysis makes it very difficult to determine accurate statistical significance. It also explains why Bem offers specific hypotheses that seem odd a priori, such as erotic images having a greater precognitive effect. Constructing hypotheses from the same data range used to test those hypotheses is a classic example of the Texas sharpshooter fallacy
"
MY RESPONSE
As WM note in their paper, this is actually how science is ordinarily done; Bem is just being honest and direct about it. Scientists typically run many exploratory experiments before finding the ones with results interesting enough to publish.
It's a meaningful point, and a reminder that science as typically practiced does not match some of the more naive notions of "scientific methodology". But it would also be impossibly cumbersome and expensive to follow the naive notion of scientific methodology and avoid exploratory work altogether, in psi or any other domain.
Ultimately this complaint against Bem's results is just another version of the "file drawer effect" hypothesis, which has been analyzed in great deal in the psi literature via meta-analyses across many experiments. The file drawer effect argument seems somewhat compelling when you look at a single experiment-set like Bem's, and becomes much less compelling when you look across the scope of all psi experiments reported, because the conclusion becomes that you'd need a huge number of carefully-run, unreported experiments to explain the total body of data.
BTW, the finding that erotic pictures give more precognitive response than other random pictures, doesn't seem terribly surprising, given the large role that sexuality plays in human psychology and evolution. If the finding were that pictures of cheese give more precognitive response than anything else, that would be more strange and surprising to me.
POINT 2
"
The paper uses the fallacy of the transposed conditional to make the case for psi powers. Essentially mixing up the difference between the probability of data given a hypothesis versus the probability of a hypothesis given data.
"
MY RESPONSE
This is a pretty silly criticism, much less worthy than the other points raised in the WM paper. Basically, when you read the discussion backing up this claim, the authors are saying that one should take into account the low a priori probability of psi in analyzing the data. OK, well ... one could just as well argue for taking to account the high a priori probability of psi given the results of prior meta-analyses or anecdotal reports of psi. Blehh.
Using the term "fallacy" here makes it seem, to people who just skim the WM paper or read only the abstract, as if Bem made some basic reasoning mistake. Yet when you actually read the WM paper, that is not what is being claimed. Rather they admit that he is following ordinary scientific methodology.
POINT 3
"
Wagenmakers' analysis of the data using a Bayesian t-test removes the significant effects claimed by Bem.
"
This is the most worthwhile point raised in the Wagenmakers et al paper.
Using a different sort of statistical test than Bem used, they re-analyze Bem's data and they find that, while the results are positive, they are not positive enough to pass the level of "statistical significance." They conclude that a somewhat larger sample size would be needed to conclude statistical significance using the test they used.
The question then becomes why to choose one statistical test over another. Indeed, it's common scientific practice to choose a statistical test that makes one's results appear significant, rather than others that do not. This is not peculiar to psi research, it's simply how science is typically done.
Near the end of their paper, WM point out that Bem's methodology is quite typical of scientific psychology research, and in fact more rigorous than most psychology papers published in good journals. What they don't note, but could have is that the same sort of methodology is used in pretty much every area of science.
They then make a series of suggestions regarding how psi research should be conducted, which would indeed increase the rigor of the research, but which a) are not followed in any branch of science, and b) would make psi research sufficiently cumbersome and expensive as to be almost impossible to conduct.
I didn't dig into the statistics deeply enough to assess the appropriateness of the particular test that WM applied (leading to their conclusion that Bem's results don't show statistical significance, for most of his experiments).
However, I am quite sure that if one applied this same Bayesian t-test to a meta-analysis over the large body of published psi experiments, one would get highly significant results. But then WM would likely raise other issues with the meta-analysis (e.g. the file drawer effect again).
Conclusion
I'll be curious to see the next part of the discussion, in which a psi-friendly statistician like Jessica Utts (or a statistician with no bias on the matter, but unbiased individuals seem very hard to come by where psi is concerned) discusses the appropriateness of WM's re-analysis of the data.
But until that, let's be clear on what WM have done. Basically, they've
The practical consequence of their latter point is that, if Bem's same experiments were done again with the same sort of results as obtained so far, then eventually a sufficient sample size would be accumulated to demonstrate significance according to WM's suggested test.
So when you peel away the rhetoric, what the WM critique really comes down to is: "Yes, his results look positive, but to pass the stricter statistical tests we suggest, one would need a larger sample size."
Of course, there is plenty of arbitrariness in our conventional criteria of significance anyway -- why do we like .05 so much, instead of .03 or .07?
So I really don't see too much meat in WM's criticism. Everyone wants to see replications of the experiments anyway, and no real invalidity in Bem's experiments, results or analyses was demonstrated.... The point made is merely that a stricter measure of significance would render these results (and an awful lot of other scientific results) insignificant until replication on a larger sample size was demonstrated. Which is an OK point -- but I'm still sorta curious to see a more careful, less obviously biased analysis of which is the best significance test to use in this case.
Recently, some psi skeptics (Wagenmakers et al) have written a technical article disputing the validity of Bem's analyses of his data.
In this blog post I'll give my reaction to the Wagenmakers et al (WM from here on) paper.
It's a frustrating paper, because it makes some valid points -- yet it also confuses the matter by inappropriately accusing Bem of committing "fallacies" and by arguing that the authors' preconceptions against psi should be used to bias the data analysis.
The paper makes 3 key points, which I will quote in the form summarized here and then respond to one by one
POINT 1
"
Bem has published his own research methodology and encourages the formulation of hypotheses after data analysis. This form of post-hoc analysis makes it very difficult to determine accurate statistical significance. It also explains why Bem offers specific hypotheses that seem odd a priori, such as erotic images having a greater precognitive effect. Constructing hypotheses from the same data range used to test those hypotheses is a classic example of the Texas sharpshooter fallacy
"
MY RESPONSE
As WM note in their paper, this is actually how science is ordinarily done; Bem is just being honest and direct about it. Scientists typically run many exploratory experiments before finding the ones with results interesting enough to publish.
It's a meaningful point, and a reminder that science as typically practiced does not match some of the more naive notions of "scientific methodology". But it would also be impossibly cumbersome and expensive to follow the naive notion of scientific methodology and avoid exploratory work altogether, in psi or any other domain.
Ultimately this complaint against Bem's results is just another version of the "file drawer effect" hypothesis, which has been analyzed in great deal in the psi literature via meta-analyses across many experiments. The file drawer effect argument seems somewhat compelling when you look at a single experiment-set like Bem's, and becomes much less compelling when you look across the scope of all psi experiments reported, because the conclusion becomes that you'd need a huge number of carefully-run, unreported experiments to explain the total body of data.
BTW, the finding that erotic pictures give more precognitive response than other random pictures, doesn't seem terribly surprising, given the large role that sexuality plays in human psychology and evolution. If the finding were that pictures of cheese give more precognitive response than anything else, that would be more strange and surprising to me.
POINT 2
"
The paper uses the fallacy of the transposed conditional to make the case for psi powers. Essentially mixing up the difference between the probability of data given a hypothesis versus the probability of a hypothesis given data.
"
MY RESPONSE
This is a pretty silly criticism, much less worthy than the other points raised in the WM paper. Basically, when you read the discussion backing up this claim, the authors are saying that one should take into account the low a priori probability of psi in analyzing the data. OK, well ... one could just as well argue for taking to account the high a priori probability of psi given the results of prior meta-analyses or anecdotal reports of psi. Blehh.
Using the term "fallacy" here makes it seem, to people who just skim the WM paper or read only the abstract, as if Bem made some basic reasoning mistake. Yet when you actually read the WM paper, that is not what is being claimed. Rather they admit that he is following ordinary scientific methodology.
POINT 3
"
Wagenmakers' analysis of the data using a Bayesian t-test removes the significant effects claimed by Bem.
"
This is the most worthwhile point raised in the Wagenmakers et al paper.
Using a different sort of statistical test than Bem used, they re-analyze Bem's data and they find that, while the results are positive, they are not positive enough to pass the level of "statistical significance." They conclude that a somewhat larger sample size would be needed to conclude statistical significance using the test they used.
The question then becomes why to choose one statistical test over another. Indeed, it's common scientific practice to choose a statistical test that makes one's results appear significant, rather than others that do not. This is not peculiar to psi research, it's simply how science is typically done.
Near the end of their paper, WM point out that Bem's methodology is quite typical of scientific psychology research, and in fact more rigorous than most psychology papers published in good journals. What they don't note, but could have is that the same sort of methodology is used in pretty much every area of science.
They then make a series of suggestions regarding how psi research should be conducted, which would indeed increase the rigor of the research, but which a) are not followed in any branch of science, and b) would make psi research sufficiently cumbersome and expensive as to be almost impossible to conduct.
I didn't dig into the statistics deeply enough to assess the appropriateness of the particular test that WM applied (leading to their conclusion that Bem's results don't show statistical significance, for most of his experiments).
However, I am quite sure that if one applied this same Bayesian t-test to a meta-analysis over the large body of published psi experiments, one would get highly significant results. But then WM would likely raise other issues with the meta-analysis (e.g. the file drawer effect again).
Conclusion
I'll be curious to see the next part of the discussion, in which a psi-friendly statistician like Jessica Utts (or a statistician with no bias on the matter, but unbiased individuals seem very hard to come by where psi is concerned) discusses the appropriateness of WM's re-analysis of the data.
But until that, let's be clear on what WM have done. Basically, they've
- raised the tired old, oft-refuted spectre of the file drawer effect, using a different verbiage from usual
- argued that one should analyze psi data using an a priori bias against it (and accused Bem of "fallacious" reasoning for not doing so)
- pointed out that if one uses a different statistical test than Bem did [though not questioning the validity of the statistical test Bem did use], one finds that his results, while positive, fall below the standard of statistical significance in most of his experiments
The practical consequence of their latter point is that, if Bem's same experiments were done again with the same sort of results as obtained so far, then eventually a sufficient sample size would be accumulated to demonstrate significance according to WM's suggested test.
So when you peel away the rhetoric, what the WM critique really comes down to is: "Yes, his results look positive, but to pass the stricter statistical tests we suggest, one would need a larger sample size."
Of course, there is plenty of arbitrariness in our conventional criteria of significance anyway -- why do we like .05 so much, instead of .03 or .07?
So I really don't see too much meat in WM's criticism. Everyone wants to see replications of the experiments anyway, and no real invalidity in Bem's experiments, results or analyses was demonstrated.... The point made is merely that a stricter measure of significance would render these results (and an awful lot of other scientific results) insignificant until replication on a larger sample size was demonstrated. Which is an OK point -- but I'm still sorta curious to see a more careful, less obviously biased analysis of which is the best significance test to use in this case.
Sunday, November 21, 2010
The Turing Church, Religion 2.0, and the Mystery of Consciousness
It was my pleasure to briefly participate in Giulio Prisco's Turing Church Online Workshop 1, on Saturday November 20 2010 in Teleplace -- a wonderfully wacky and wide-ranging exploration of transhumanist spirituality and “Religion 2.0.″
The video proceedings are here.
I didn't participate in the whole workshop since it was a busy day for me, I just logged on briefly to give a talk and answer some questions. But I found the theme quite fascinating.
Giulio said I should assume the participants were already basically familiar with my thinking on transhumanist spirituality as expressed in my little book A Cosmist Manifesto that I wrote earlier this year, and he asked me to venture in some slightly different direction. I'm not sure I fulfilled that request all that well, but anyway, I'll paste here the notes I wrote as a basis for my talk in the workshop. I didn't read these notes with any precision, so if you want to know what I actually said you'll have to watch the video; but the talk was a more informal improvisation on the same basic theme...
"The relation between transhumanism and spirituality is a big topic, which I've thought about a lot -- right now I'll just make a few short comments. Sorry that I won't be able to stick around for this whole meeting today, I have some family stuff I need to do, but I'm happy to be able to participate at least briefly by saying a few remarks.
"Earlier this year I wrote a book touching on some of these comments, called "A Cosmist Manifesto" -- I'm not going to reiterate all that material now, just touch on a few key points.
"The individual human mind has a tendency to tie itself in what the psychologist Stanislaw Grof calls "knots" -- intricate webs of self-contradiction and fear, that cause emotional pain and cognitive confusion and serve as traps for mental energy. Ultimately these knots are largely rooted in the human self's fear of losing itself --- the self's fear of realizing that it lacks fundamental reality, and is basically a construct whose main goals are to keep the body going and reproducing and to preserve itself. These are some complicated words for describing something pretty basic, but I guess we all know what I'm talking about.
"And then there are the social knots, going beyond the individual ones… the knots we tie each other up in…
"These knots are serious problems for all of us -- and they're an even more serious problem when you think about the potential consequences of advanced technology in the next decade. We're on the verge of creating superhuman AI and molecular nanotech and brain-computer interfacing and so forth -- but we're still pretty much fucked up with psychological and social confusions! As Freud pointed out in Civilization and its Discontents, we're largely operating with motivational systems evolved for being hunter-gatherers in the African savannah, but the world we're creating for ourselves is dramatically different from that.
"Human society has come up with a bunch of different ways to get past these knots.
"One of them is religion -- which opens a doorway to transpersonal experience, going beyond self and society, opening things up to a broader domain of perceiving, being, understanding and acting. If you're not familiar with more philosophical side of the traditional religions you should look at Aldous Huxley's classic book "The Perennial Philosophy" -- it was really an eye-opener for me.
"Another method for getting past the knots is science. By focusing on empirical data, collectively perceived and understood, science lets us go beyond our preconceptions and emotions and biases and ideas. Science, with its focus on data and collective rational understanding, provides a powerful engine for growth of understanding. There's a saying that "science advances one funeral at a time" -- i.e. old scientific ideas only die when their proponents die. But the remarkable thing is, this isn't entirely true. Science has an amazing capability to push people to give up their closely held ideas, when these ideas don't mesh well with the evidence.
"What I see in the transhumanism-meets-spirituality connection is the possibility of somehow bringing together these two great ways of getting beyond the knots. If science and spirituality can come together somehow, we may have a much more powerful way of getting past the individual and social knots that bind us. If we could somehow combine the rigorous data focus of science with the personal and collective mind-purification of spiritual traditions, then we'd have something pretty new and pretty interesting -- and maybe something that could help us grapple with the complex issues modern technology is going to bring us in the next few decades
"One specific area of science that seems very relevant to these considerations is consciousness studies. Science is having a hard time grappling with consciousness, though it's discovering a lot about neural and cognitive correlates of consciousness. Spiritual traditions have discovered a lot about consciousness, though a lot of this knowledge is expressed in language that's hard for modern people to deal with. I wonder if some kind of science plus spirituality hybrid could provide a new way for groups of people to understand consciousness, combining scientific data and spiritual understanding.
"One idea I mentioned in the Cosmist Manifesto book is some sort of "Confederation of Cosmists", and Giulio asked me to say a little bit about that here. The core idea is obvious -- some kind of social group of individuals interested in both advanced technology and its implications, and personal growth and mind-expansion. The specific manifestation of the idea isn't too clear. But I wonder if one useful approach might be to focus on the cross-disciplinary understanding of consciousness -- using science and spirituality, and also advanced technologies like neuroscience and BCI and AGI. My thinking is that consciousness studies is one concrete area that truly seems to demand some kind of fusion of scientific and spiritual ideas … so maybe focusing on that in a truly broad, cross-tradition, Cosmist way could help us come together more and over help us work together to overcome our various personal and collective knots, and build a better future, and all that good stuff….
"Anyway there are just some preliminary thoughts, these are things I'm thinking about a lot these days, and I look forward to sharing my ideas more with you as my thoughts develop -- and I'll be catching the rest of this conference via the video recordings later on."
Fun stuff to think about -- though I don't have too much time for it these days, as my AGI and bioinformatics work seems to be taking all my time. But at some future point, I really do think the cross-disciplinary introspective/scientific individual/collective investigation of consciousness is well worth devoting attention to, and is going to bear some pretty fascinating fruit....
The video proceedings are here.
I didn't participate in the whole workshop since it was a busy day for me, I just logged on briefly to give a talk and answer some questions. But I found the theme quite fascinating.
Giulio said I should assume the participants were already basically familiar with my thinking on transhumanist spirituality as expressed in my little book A Cosmist Manifesto that I wrote earlier this year, and he asked me to venture in some slightly different direction. I'm not sure I fulfilled that request all that well, but anyway, I'll paste here the notes I wrote as a basis for my talk in the workshop. I didn't read these notes with any precision, so if you want to know what I actually said you'll have to watch the video; but the talk was a more informal improvisation on the same basic theme...
"The relation between transhumanism and spirituality is a big topic, which I've thought about a lot -- right now I'll just make a few short comments. Sorry that I won't be able to stick around for this whole meeting today, I have some family stuff I need to do, but I'm happy to be able to participate at least briefly by saying a few remarks.
"Earlier this year I wrote a book touching on some of these comments, called "A Cosmist Manifesto" -- I'm not going to reiterate all that material now, just touch on a few key points.
"The individual human mind has a tendency to tie itself in what the psychologist Stanislaw Grof calls "knots" -- intricate webs of self-contradiction and fear, that cause emotional pain and cognitive confusion and serve as traps for mental energy. Ultimately these knots are largely rooted in the human self's fear of losing itself --- the self's fear of realizing that it lacks fundamental reality, and is basically a construct whose main goals are to keep the body going and reproducing and to preserve itself. These are some complicated words for describing something pretty basic, but I guess we all know what I'm talking about.
"And then there are the social knots, going beyond the individual ones… the knots we tie each other up in…
"These knots are serious problems for all of us -- and they're an even more serious problem when you think about the potential consequences of advanced technology in the next decade. We're on the verge of creating superhuman AI and molecular nanotech and brain-computer interfacing and so forth -- but we're still pretty much fucked up with psychological and social confusions! As Freud pointed out in Civilization and its Discontents, we're largely operating with motivational systems evolved for being hunter-gatherers in the African savannah, but the world we're creating for ourselves is dramatically different from that.
"Human society has come up with a bunch of different ways to get past these knots.
"One of them is religion -- which opens a doorway to transpersonal experience, going beyond self and society, opening things up to a broader domain of perceiving, being, understanding and acting. If you're not familiar with more philosophical side of the traditional religions you should look at Aldous Huxley's classic book "The Perennial Philosophy" -- it was really an eye-opener for me.
"Another method for getting past the knots is science. By focusing on empirical data, collectively perceived and understood, science lets us go beyond our preconceptions and emotions and biases and ideas. Science, with its focus on data and collective rational understanding, provides a powerful engine for growth of understanding. There's a saying that "science advances one funeral at a time" -- i.e. old scientific ideas only die when their proponents die. But the remarkable thing is, this isn't entirely true. Science has an amazing capability to push people to give up their closely held ideas, when these ideas don't mesh well with the evidence.
"What I see in the transhumanism-meets-spirituality connection is the possibility of somehow bringing together these two great ways of getting beyond the knots. If science and spirituality can come together somehow, we may have a much more powerful way of getting past the individual and social knots that bind us. If we could somehow combine the rigorous data focus of science with the personal and collective mind-purification of spiritual traditions, then we'd have something pretty new and pretty interesting -- and maybe something that could help us grapple with the complex issues modern technology is going to bring us in the next few decades
"One specific area of science that seems very relevant to these considerations is consciousness studies. Science is having a hard time grappling with consciousness, though it's discovering a lot about neural and cognitive correlates of consciousness. Spiritual traditions have discovered a lot about consciousness, though a lot of this knowledge is expressed in language that's hard for modern people to deal with. I wonder if some kind of science plus spirituality hybrid could provide a new way for groups of people to understand consciousness, combining scientific data and spiritual understanding.
"One idea I mentioned in the Cosmist Manifesto book is some sort of "Confederation of Cosmists", and Giulio asked me to say a little bit about that here. The core idea is obvious -- some kind of social group of individuals interested in both advanced technology and its implications, and personal growth and mind-expansion. The specific manifestation of the idea isn't too clear. But I wonder if one useful approach might be to focus on the cross-disciplinary understanding of consciousness -- using science and spirituality, and also advanced technologies like neuroscience and BCI and AGI. My thinking is that consciousness studies is one concrete area that truly seems to demand some kind of fusion of scientific and spiritual ideas … so maybe focusing on that in a truly broad, cross-tradition, Cosmist way could help us come together more and over help us work together to overcome our various personal and collective knots, and build a better future, and all that good stuff….
"Anyway there are just some preliminary thoughts, these are things I'm thinking about a lot these days, and I look forward to sharing my ideas more with you as my thoughts develop -- and I'll be catching the rest of this conference via the video recordings later on."
Fun stuff to think about -- though I don't have too much time for it these days, as my AGI and bioinformatics work seems to be taking all my time. But at some future point, I really do think the cross-disciplinary introspective/scientific individual/collective investigation of consciousness is well worth devoting attention to, and is going to bear some pretty fascinating fruit....
Friday, October 29, 2010
The Singularity Institute's Scary Idea (and Why I Don't Buy It)
I recently wrote a blog post about my own AI project, but it attracted a bunch of adversarial comments from folks influenced by the Singularity Institute for AI's (rather different) perspective on the best approach to AI R&D. I responded to some of these comments there.
(Quick note for those who don't know: the Singularity Institute for AI is not affiliated with Singularity University, though there are some overlaps ... Ray Kurzweil is an Advisor to the former and the founder of the latter; and I am an Advisor to both.)
Following that discussion, a bunch of people have emailed me in the last couple weeks asking me to write something clearly and specifically addressing my views on SIAI's perspective on the future of AI. I don't want to spend a lot of time on this but I decided to bow to popular demand and write a blog post...
Of course, there are a lot of perspectives in the world that I don't agree with, and I don't intend to write blog posts explaining the reasons for my disagreement with all of them! But since I've had some involvement with SIAI in the past, I guess it's sort of a special case.
First of all I want to clarify I'm not in disagreement with the existence of SIAI as an institution, nor with the majority of their activities -- only with certain positions habitually held by some SIAI researchers, and by the community of individuals heavily involved with SIAI. And specifically with a particular line of thinking that I'll refer to here as "SIAI's Scary Idea."
Roughly, the Scary Idea posits that: If I or anybody else actively trying to build advanced AGI succeeds, we're highly likely to cause an involuntary end to the human race.
Brief Digression: My History with SIAI
Before getting started with the meat of the post, I'll give a few more personal comments, to fill in some history for those readers who don't know it, or who know only parts. Readers who are easily bored may wish to skip to the next section,
SIAI has been quite good to me, overall. I've enjoyed all the Singularity Summits, which they've hosted, very much; I think they've played a major role in the advancement of society's thinking about the future, and I've felt privileged to speak at them. And I applaud SIAI for consistently being open to Summit speakers whose views are strongly divergent from those commonly held in the SIAI community.
Also, in 2008, SIAI and my company Novamente LLC seed-funded the OpenCog open-source AGI project (based on software code spun out from Novamente). The SIAI/OpenCog relationship diminished substantially when Tyler Emerson passed the leadership of SIAI along to Michael Vassar, but it was instrumental in getting OpenCog off the ground. I've also enjoyed working with Michael Vassar on the Board of Humanity+, of which I'm Chair and he's a Board member.
When SIAI was helping fund OpenCog, I took the title of "Director of Research" of SIAI, but I never actually directed any research there apart from OpenCog. The other SIAI research was always directed by others, which was fine with me. There were occasional discussions about operating in a more unified manner, but it didn't happen. All this is perfectly ordinary in a small start-up type organization.
Once SIAI decided OpenCog was no longer within its focus, after a bit of delay I decided it didn't make sense for me to hold the Director of Research title anymore, since as things were evolving, I wasn't directing any SIAI research. I remain as an Advisor to SIAI, which is going great.
Now, on to the meat of the post….
SIAI's Scary Idea (Which I Don't Agree With)
SIAI's leaders and community members have a lot of beliefs and opinions, many of which I share and many not, but the key difference between our perspectives lies in what I'll call SIAI's "Scary Idea", which is the idea that: progressing toward advanced AGI without a design for "provably non-dangerous AGI" (or something closely analogous, often called "Friendly AI" in SIAI lingo) is highly likely to lead to an involuntary end for the human race.
(SIAI's Scary Idea has been worded in many different ways by many different people, and I tried in the above paragraph to word it in a way that captures the idea fairly if approximatively, and won't piss off too many people.)
Of course it's rarely clarified what "provably" really means. A mathematical proof can only be applied to the real world in the context of some assumptions, so maybe "provably non-dangerous AGI" means "an AGI whose safety is implied by mathematical arguments together with assumptions that are believed reasonable by some responsible party"? (where the responsible party is perhaps "the overwhelming majority of scientists" … or SIAI itself?)….. I'll say a little more about this a bit below.
Please note that, although I don't agree with the Scary Idea, I do agree that the development of advanced AGI has significant risks associated with it. There are also dramatic potential benefits associated with it, including the potential of protection against risks from other technologies (like nanotech, biotech, narrow AI, etc.). So the development of AGI has difficult cost-benefit balances associated with it -- just like the development of many other technologies.
I also agree with Nick Bostrom and a host of SF writers and many others that AGI is a potential "existential risk" -- i.e. that in the worst case, AGI could wipe out humanity entirely. I think nanotech and biotech and narrow AI could also do so, along with a bunch of other things.
I certainly don't want to see the human race wiped out! I personally would like to transcend the legacy human condition and become a transhuman superbeing … and I would like everyone else to have the chance to do so, if they want to. But even though I think this kind of transcendence will be possible, and will be desirable to many, I wouldn't like to see anyone forced to transcend in this way. I would like to see the good old fashioned human race continue, if there are humans who want to maintain their good old fashioned humanity, even if other options are available
But SIAI's Scary Idea goes way beyond the mere statement that there are risks as well as benefits associated with advanced AGI, and that AGI is a potential existential risk.
Finally, I note that most of the other knowledgeable futurist scientists and philosophers, who have come into close contact with SIAI's perspective, also don't accept the Scary Idea. Examples include Robin Hanson, Nick Bostrom and Ray Kurzweil.
There's nothing wrong with having radical ideas that one's respected peers mostly don't accept. I totally get that: My own approach to AGI is somewhat radical, and most of my friends in the AGI research community, while they respect my work and see its potential, aren't quite as enthused about it as I am. Radical positive changes are often brought about by people who clearly understand certain radical ideas well before anyone else "sees the light." However, my own radical ideas are not telling whole research fields that if they succeed they're bound to kill everybody ... so it's a somewhat different situation.
What is the Argument for the Scary Idea?
Although an intense interest in rationalism is one of the hallmarks of the SIAI community, still I have not yet seen a clear logical argument for the Scary Idea laid out anywhere. (If I'm wrong, please send me the link, and I'll revise this post accordingly. Be aware that I've already at least skimmed everything Eliezer Yudkowsky has written on related topics.)
So if one wants a clear argument for the Scary Idea, one basically has to construct it oneself.
As far as I can tell from discussions and the available online material, some main ingredients of peoples' reasons for believing the Scary Idea are ideas like:
If you put the above points all together, you come up with a heuristic argument for the Scary Idea. Roughly, the argument goes something like: If someone builds an advanced AGI without a provably Friendly architecture, probably it will have a hard takeoff, and then probably this will lead to a superhuman AGI system with an architecture drawn from the vast majority of mind-architectures that are not sufficiently harmonious with the complex, fragile human value system to make humans happy and keep humans around.
The line of argument makes sense, if you accept the premises.
But, I don't.
I think the first of the above points is reasonably plausible, though I'm not by any means convinced. I think the relation between breadth of intelligence and depth of empathy is a subtle issue which none of us fully understands (yet). It's possible that with sufficient real-world intelligence tends to come a sense of connectedness with the universe that militates against squashing other sentiences. But I'm not terribly certain of this, any more than I'm terribly certain of its opposite.
I agree much less with the final three points listed above. And I haven't seen any careful logical arguments for these points.
I doubt human value is particularly fragile. Human value has evolved and morphed over time and will continue to do so. It already takes multiple different forms. It will likely evolve in future in coordination with AGI and other technology. I think it's fairly robust.
I think a hard takeoff is possible, though I don't know how to estimate the odds of one occurring with any high confidence. I think it's very unlikely to occur until we have an AGI system that has very obviously demonstrated general intelligence at the level of a highly intelligent human. And I think the path to this "hard takeoff enabling" level of general intelligence is going to be somewhat gradual, not extremely sudden.
I don't have any strong sense of the probability of a hard takeoff, from an apparently but not provably human-friendly AGI, leading to an outcome likable to humans. I suspect this probability depends on many features of the AGI, which we will identify over the next years & decades via theorizing based on the results of experimentation with early-stage AGIs.
Yes, you may argue: the Scary Idea hasn't been rigorously shown to be true… but what if it IS true?
OK but ... pointing out that something scary is possible, is a very different thing from having an argument that it's likely.
The Scary Idea is certainly something to keep in mind, but there are also many other risks to keep in mind, some much more definite and palpable. Personally, I'm a lot more worried about nasty humans taking early-stage AGIs and using them for massive destruction, than about speculative risks associated with little-understood events like hard takeoffs.
Is Provably Safe or "Friendly" AGI A Feasible Idea?
The Scary Idea posits that if someone creates advanced AGI that isn't somehow provably safe, it's almost sure to kill us all.
But not only am I unconvinced of this, I'm also quite unconvinced that "provably safe" AGI is even feasible.
The idea of provably safe AGI is typically presented as something that would exist within mathematical computation theory or some variant thereof. So that's one obvious limitation of the idea: mathematical computers don't exist in the real world, and real-world physical computers must be interpreted in terms of the laws of physics, and humans' best understanding of the "laws" of physics seems to radically change from time to time. So even if there were a design for provably safe real-world AGI, based on current physics, the relevance of the proof might go out the window when physics next gets revised.
Also, there are always possibilities like: the alien race that is watching us and waiting for us to achieve an IQ of 333, at which point it will swoop down upon us and eat us, or merge with us. We can't rule this out via any formal proof, and we can't meaningfully estimate the odds of it either. Yes, this sounds science-fictional and outlandish; but is it really more outlandish and speculative than the Scary Idea?
A possibility that strikes me as highly likely is that, once we have created advanced AGI and have linked our brains with it collectively, most of our old legacy human ideas (including physical law, aliens, and Friendly AI) will seem extremely limited and ridiculous.
Another issue is that the goal of "Friendliness to humans" or "safety" or whatever you want to call it, is rather nebulous and difficult to pin down. Science fiction has explored this theme extensively. So even if we could prove something about "smart AGI systems with a certain architecture that are guaranteed to achieve goal G," it might be infeasible to apply this to make AGI systems that are safe in the real-world -- simply because we don't know how to boil down the everyday intuitive notions of "safety" or "Friendliness" into a mathematically precise goal G like the proof refers to.
This is related to the point Eliezer Yudkowsky makes that "value is complex" -- actually, human value is not only complex, it's nebulous and fuzzy and ever-shifting, and humans largely grok it by implicit procedural, empathic and episodic knowledge rather than explicit declarative or linguistic knowledge. Transmitting human values to an AGI is likely to be best done via interacting with the AGI in real life, but this is not the sort of process that readily lends itself to guarantees or formalization.
Eliezer has suggested a speculative way of getting human values into AGI systems called Coherent Extrapolated Volition, but I think this is a very science-fictional and incredibly infeasible idea (though a great SF notion). I've discussed it and proposed some possibly more realistic alternatives in a previous blog post (e.g. a notion called Coherent Aggregated Volition). But my proposed alternatives aren't guaranteed-to-succeed nor neatly formalized.
But setting those worries aside, is the computation-theoretic version of provably safe AI even possible? Could one design an AGI system and prove in advance that, given certain reasonable assumptions about physics and its environment, it would never veer too far from its initial goal (e.g. a formalized version of the goal of treating humans safely, or whatever)?
I very much doubt one can do so, except via designing a fictitious AGI that can't really be implemented because it uses infeasibly much computational resources. My GOLEM design, sketched in this article, seems to me a possible path to a provably safe AGI -- but it's too computationally wasteful to be practically feasible.
I strongly suspect that to achieve high levels of general intelligence using realistically limited computational resources, one is going to need to build systems with a nontrivial degree of fundamental unpredictability to them. This is what neuroscience suggests, it's what my concrete AGI design work suggests, and it's what my theoretical work on GOLEM and related ideas suggests. And none of the public output of SIAI researchers or enthusiasts has given me any reason to believe otherwise, yet.
Practical Implications
The above discussion of SIAI's Scary Idea may just sound like fun science-fictional speculation -- but the reason I'm writing this blog post is that when I posted a recent blog post about my current AGI project, the comments field got swamped with SIAI-influenced people saying stuff in the vein of: Creating an AGI without a proof of Friendliness is essentially equivalent to killing all people! So I really hope your OpenCog work fails, so you don't kill everybody!!!
(One amusing/alarming quote from a commentator (probably not someone directly affiliated with SIAI) was "if you go ahead with an AGI when you're not 100% sure that it's safe, you're committing the Holocaust." But it wasn't just one extreme commentator, it was a bunch … and then a bunch of others commenting to me privately via email.)
If one fully accepts SIAI's Scary Idea, then one should not work on practical AGI projects, nor should one publish papers on the theory of how to build AGI systems. Instead, one should spend one's time trying to figure out an AGI design that is somehow provable-in-advance to be a Good Guy. For this reason, SIAI's research group is not currently trying to do any practical AGI work.
Actually, so far as I know, my "GOLEM" AGI design (mentioned above) is closer to a "provably Friendly AI" than anything the SIAI research team has come up with. At least, it's closer than anything they have made public.
However GOLEM is not something that could be practically implemented in the near future. It's horribly computationally inefficient, compared to a real-world AGI design like the OpenCog system I'm now working on (with many others -- actually I'm doing very little programming these days, so happily the project is moving forward with the help of others on the software design and coding side, while I contribute at the algorithm, math, design, theory, management and fundraising levels).
I agree that AGI ethics is a Very Important Problem. But I doubt the problem is most effectively addressed by theory alone. I think the way to come to a useful real-world understanding of AGI ethics is going to be to
Nothing, really -- if you hold the views of most AI researchers or futurists. There are plenty of disagreements about the right path to AGI, but wide and implicit agreement that something like the above path is sensible.
But, if you adhere to SIAI's Scary Idea, there's a big problem with this approach -- because, according to the Scary Idea, there's too huge of a risk that these early-stage AGI systems are going to experience a hard takeoff and self-modify into something that will destroy us all.
But I just don't buy the Scary Idea.
I do see a real risk that, if we proceed in the manner I'm advocating, some nasty people will take the early-stage AGIs and either use them for bad ends, or proceed to hastily create a superhuman AGI that then does bad things of its own volition. These are real risks that must be thought about hard, and protected against as necessary. But they are different from the Scary Idea. And they are not so different from the risks implicit in a host of other advanced technologies.
Conclusion
So, there we go.
I think SIAI is performing a useful service by helping bring these sorts of ideas to the attention of the futurist community (alongside the other services they're performing, like the wonderful Singularity Summits). But, that said, I think the Scary Idea is potentially a harmful one. At least, it WOULD be a harmful one, if more people believed it; so I'm glad it's currently restricted to a rather small subset of the futurist community.
Many people die each day, and many others are miserable for various reasons -- and all sorts of other advanced and potentially dangerous technologies are currently under active development. My own view is that unaided human minds may well be unable to deal with the complexity and risk of the world that human technology is unleashing. I actually suspect that our best hope for survival and growth through the 21st century is to create advanced AGIs to help us on our way -- to cure disease, to develop nanotech and better AGI and invent new technologies; and to help us keep nasty people from doing destructive things with advanced technology.
I think that to avoid actively developing AGI, out of speculative concerns like the Scary Idea, would be an extremely bad idea.
That is, rather than "if you go ahead with an AGI when you're not 100% sure that it's safe, you're committing the Holocaust," I suppose my view is closer to "if you avoid creating beneficial AGI because of speculative concerns, then you're killing my grandma" !! (Because advanced AGI will surely be able to help us cure human diseases and vastly extend and improve human life.)
So perhaps I could adopt the slogan: "You don't have to kill my grandma to avoid the Holocaust!" … but really, folks… Well, you get the point….
Humanity is on a risky course altogether, but no matter what I decide to do with my life and career (and no matter what Bill Joy or Jaron Lanier or Bill McKibben, etc., write), the race is not going to voluntarily halt technological progress. It's just not happening.
We just need to accept the risk, embrace the thrill of the amazing time we were born into, and try our best to develop near-inevitable technologies like AGI in a responsible and ethical way.
And to me, responsible AGI development doesn't mean fixating on speculative possible dangers and halting development until ill-defined, likely-unsolvable theoretical/philosophical issues are worked out to everybody's (or some elite group's) satisfaction.
Rather, it means proceeding with the work carefully and openly, learning what we can as we move along -- and letting experiment and theory grow together ... as they have been doing quite successfully for the last few centuries, at a fantastically accelerating pace.
And so it goes.
(Quick note for those who don't know: the Singularity Institute for AI is not affiliated with Singularity University, though there are some overlaps ... Ray Kurzweil is an Advisor to the former and the founder of the latter; and I am an Advisor to both.)
Following that discussion, a bunch of people have emailed me in the last couple weeks asking me to write something clearly and specifically addressing my views on SIAI's perspective on the future of AI. I don't want to spend a lot of time on this but I decided to bow to popular demand and write a blog post...
Of course, there are a lot of perspectives in the world that I don't agree with, and I don't intend to write blog posts explaining the reasons for my disagreement with all of them! But since I've had some involvement with SIAI in the past, I guess it's sort of a special case.
First of all I want to clarify I'm not in disagreement with the existence of SIAI as an institution, nor with the majority of their activities -- only with certain positions habitually held by some SIAI researchers, and by the community of individuals heavily involved with SIAI. And specifically with a particular line of thinking that I'll refer to here as "SIAI's Scary Idea."
Roughly, the Scary Idea posits that: If I or anybody else actively trying to build advanced AGI succeeds, we're highly likely to cause an involuntary end to the human race.
Brief Digression: My History with SIAI
Before getting started with the meat of the post, I'll give a few more personal comments, to fill in some history for those readers who don't know it, or who know only parts. Readers who are easily bored may wish to skip to the next section,
SIAI has been quite good to me, overall. I've enjoyed all the Singularity Summits, which they've hosted, very much; I think they've played a major role in the advancement of society's thinking about the future, and I've felt privileged to speak at them. And I applaud SIAI for consistently being open to Summit speakers whose views are strongly divergent from those commonly held in the SIAI community.
Also, in 2008, SIAI and my company Novamente LLC seed-funded the OpenCog open-source AGI project (based on software code spun out from Novamente). The SIAI/OpenCog relationship diminished substantially when Tyler Emerson passed the leadership of SIAI along to Michael Vassar, but it was instrumental in getting OpenCog off the ground. I've also enjoyed working with Michael Vassar on the Board of Humanity+, of which I'm Chair and he's a Board member.
When SIAI was helping fund OpenCog, I took the title of "Director of Research" of SIAI, but I never actually directed any research there apart from OpenCog. The other SIAI research was always directed by others, which was fine with me. There were occasional discussions about operating in a more unified manner, but it didn't happen. All this is perfectly ordinary in a small start-up type organization.
Once SIAI decided OpenCog was no longer within its focus, after a bit of delay I decided it didn't make sense for me to hold the Director of Research title anymore, since as things were evolving, I wasn't directing any SIAI research. I remain as an Advisor to SIAI, which is going great.
Now, on to the meat of the post….
SIAI's Scary Idea (Which I Don't Agree With)
SIAI's leaders and community members have a lot of beliefs and opinions, many of which I share and many not, but the key difference between our perspectives lies in what I'll call SIAI's "Scary Idea", which is the idea that: progressing toward advanced AGI without a design for "provably non-dangerous AGI" (or something closely analogous, often called "Friendly AI" in SIAI lingo) is highly likely to lead to an involuntary end for the human race.
(SIAI's Scary Idea has been worded in many different ways by many different people, and I tried in the above paragraph to word it in a way that captures the idea fairly if approximatively, and won't piss off too many people.)
Of course it's rarely clarified what "provably" really means. A mathematical proof can only be applied to the real world in the context of some assumptions, so maybe "provably non-dangerous AGI" means "an AGI whose safety is implied by mathematical arguments together with assumptions that are believed reasonable by some responsible party"? (where the responsible party is perhaps "the overwhelming majority of scientists" … or SIAI itself?)….. I'll say a little more about this a bit below.
Please note that, although I don't agree with the Scary Idea, I do agree that the development of advanced AGI has significant risks associated with it. There are also dramatic potential benefits associated with it, including the potential of protection against risks from other technologies (like nanotech, biotech, narrow AI, etc.). So the development of AGI has difficult cost-benefit balances associated with it -- just like the development of many other technologies.
I also agree with Nick Bostrom and a host of SF writers and many others that AGI is a potential "existential risk" -- i.e. that in the worst case, AGI could wipe out humanity entirely. I think nanotech and biotech and narrow AI could also do so, along with a bunch of other things.
I certainly don't want to see the human race wiped out! I personally would like to transcend the legacy human condition and become a transhuman superbeing … and I would like everyone else to have the chance to do so, if they want to. But even though I think this kind of transcendence will be possible, and will be desirable to many, I wouldn't like to see anyone forced to transcend in this way. I would like to see the good old fashioned human race continue, if there are humans who want to maintain their good old fashioned humanity, even if other options are available
But SIAI's Scary Idea goes way beyond the mere statement that there are risks as well as benefits associated with advanced AGI, and that AGI is a potential existential risk.
Finally, I note that most of the other knowledgeable futurist scientists and philosophers, who have come into close contact with SIAI's perspective, also don't accept the Scary Idea. Examples include Robin Hanson, Nick Bostrom and Ray Kurzweil.
There's nothing wrong with having radical ideas that one's respected peers mostly don't accept. I totally get that: My own approach to AGI is somewhat radical, and most of my friends in the AGI research community, while they respect my work and see its potential, aren't quite as enthused about it as I am. Radical positive changes are often brought about by people who clearly understand certain radical ideas well before anyone else "sees the light." However, my own radical ideas are not telling whole research fields that if they succeed they're bound to kill everybody ... so it's a somewhat different situation.
What is the Argument for the Scary Idea?
Although an intense interest in rationalism is one of the hallmarks of the SIAI community, still I have not yet seen a clear logical argument for the Scary Idea laid out anywhere. (If I'm wrong, please send me the link, and I'll revise this post accordingly. Be aware that I've already at least skimmed everything Eliezer Yudkowsky has written on related topics.)
So if one wants a clear argument for the Scary Idea, one basically has to construct it oneself.
As far as I can tell from discussions and the available online material, some main ingredients of peoples' reasons for believing the Scary Idea are ideas like:
- If one pulled a random mind from the space of all possible minds, the odds of it being friendly to humans (as opposed to, e.g., utterly ignoring us, and being willing to repurpose our molecules for its own ends) are very low
- Human value is fragile as well as complex, so if you create an AGI with a roughly-human-like value system, then this may not be good enough, and it is likely to rapidly diverge into something with little or no respect for human values
- "Hard takeoffs" (in which AGIs recursively self-improve and massively increase their intelligence) are fairly likely once AGI reaches a certain level of intelligence; and humans will have little hope of stopping these events
- A hard takeoff, unless it starts from an AGI designed in a "provably Friendly" way, is highly likely to lead to an AGI system that doesn't respect the rights of humans to exist
If you put the above points all together, you come up with a heuristic argument for the Scary Idea. Roughly, the argument goes something like: If someone builds an advanced AGI without a provably Friendly architecture, probably it will have a hard takeoff, and then probably this will lead to a superhuman AGI system with an architecture drawn from the vast majority of mind-architectures that are not sufficiently harmonious with the complex, fragile human value system to make humans happy and keep humans around.
The line of argument makes sense, if you accept the premises.
But, I don't.
I think the first of the above points is reasonably plausible, though I'm not by any means convinced. I think the relation between breadth of intelligence and depth of empathy is a subtle issue which none of us fully understands (yet). It's possible that with sufficient real-world intelligence tends to come a sense of connectedness with the universe that militates against squashing other sentiences. But I'm not terribly certain of this, any more than I'm terribly certain of its opposite.
I agree much less with the final three points listed above. And I haven't seen any careful logical arguments for these points.
I doubt human value is particularly fragile. Human value has evolved and morphed over time and will continue to do so. It already takes multiple different forms. It will likely evolve in future in coordination with AGI and other technology. I think it's fairly robust.
I think a hard takeoff is possible, though I don't know how to estimate the odds of one occurring with any high confidence. I think it's very unlikely to occur until we have an AGI system that has very obviously demonstrated general intelligence at the level of a highly intelligent human. And I think the path to this "hard takeoff enabling" level of general intelligence is going to be somewhat gradual, not extremely sudden.
I don't have any strong sense of the probability of a hard takeoff, from an apparently but not provably human-friendly AGI, leading to an outcome likable to humans. I suspect this probability depends on many features of the AGI, which we will identify over the next years & decades via theorizing based on the results of experimentation with early-stage AGIs.
Yes, you may argue: the Scary Idea hasn't been rigorously shown to be true… but what if it IS true?
OK but ... pointing out that something scary is possible, is a very different thing from having an argument that it's likely.
The Scary Idea is certainly something to keep in mind, but there are also many other risks to keep in mind, some much more definite and palpable. Personally, I'm a lot more worried about nasty humans taking early-stage AGIs and using them for massive destruction, than about speculative risks associated with little-understood events like hard takeoffs.
Is Provably Safe or "Friendly" AGI A Feasible Idea?
The Scary Idea posits that if someone creates advanced AGI that isn't somehow provably safe, it's almost sure to kill us all.
But not only am I unconvinced of this, I'm also quite unconvinced that "provably safe" AGI is even feasible.
The idea of provably safe AGI is typically presented as something that would exist within mathematical computation theory or some variant thereof. So that's one obvious limitation of the idea: mathematical computers don't exist in the real world, and real-world physical computers must be interpreted in terms of the laws of physics, and humans' best understanding of the "laws" of physics seems to radically change from time to time. So even if there were a design for provably safe real-world AGI, based on current physics, the relevance of the proof might go out the window when physics next gets revised.
Also, there are always possibilities like: the alien race that is watching us and waiting for us to achieve an IQ of 333, at which point it will swoop down upon us and eat us, or merge with us. We can't rule this out via any formal proof, and we can't meaningfully estimate the odds of it either. Yes, this sounds science-fictional and outlandish; but is it really more outlandish and speculative than the Scary Idea?
A possibility that strikes me as highly likely is that, once we have created advanced AGI and have linked our brains with it collectively, most of our old legacy human ideas (including physical law, aliens, and Friendly AI) will seem extremely limited and ridiculous.
Another issue is that the goal of "Friendliness to humans" or "safety" or whatever you want to call it, is rather nebulous and difficult to pin down. Science fiction has explored this theme extensively. So even if we could prove something about "smart AGI systems with a certain architecture that are guaranteed to achieve goal G," it might be infeasible to apply this to make AGI systems that are safe in the real-world -- simply because we don't know how to boil down the everyday intuitive notions of "safety" or "Friendliness" into a mathematically precise goal G like the proof refers to.
This is related to the point Eliezer Yudkowsky makes that "value is complex" -- actually, human value is not only complex, it's nebulous and fuzzy and ever-shifting, and humans largely grok it by implicit procedural, empathic and episodic knowledge rather than explicit declarative or linguistic knowledge. Transmitting human values to an AGI is likely to be best done via interacting with the AGI in real life, but this is not the sort of process that readily lends itself to guarantees or formalization.
Eliezer has suggested a speculative way of getting human values into AGI systems called Coherent Extrapolated Volition, but I think this is a very science-fictional and incredibly infeasible idea (though a great SF notion). I've discussed it and proposed some possibly more realistic alternatives in a previous blog post (e.g. a notion called Coherent Aggregated Volition). But my proposed alternatives aren't guaranteed-to-succeed nor neatly formalized.
But setting those worries aside, is the computation-theoretic version of provably safe AI even possible? Could one design an AGI system and prove in advance that, given certain reasonable assumptions about physics and its environment, it would never veer too far from its initial goal (e.g. a formalized version of the goal of treating humans safely, or whatever)?
I very much doubt one can do so, except via designing a fictitious AGI that can't really be implemented because it uses infeasibly much computational resources. My GOLEM design, sketched in this article, seems to me a possible path to a provably safe AGI -- but it's too computationally wasteful to be practically feasible.
I strongly suspect that to achieve high levels of general intelligence using realistically limited computational resources, one is going to need to build systems with a nontrivial degree of fundamental unpredictability to them. This is what neuroscience suggests, it's what my concrete AGI design work suggests, and it's what my theoretical work on GOLEM and related ideas suggests. And none of the public output of SIAI researchers or enthusiasts has given me any reason to believe otherwise, yet.
Practical Implications
The above discussion of SIAI's Scary Idea may just sound like fun science-fictional speculation -- but the reason I'm writing this blog post is that when I posted a recent blog post about my current AGI project, the comments field got swamped with SIAI-influenced people saying stuff in the vein of: Creating an AGI without a proof of Friendliness is essentially equivalent to killing all people! So I really hope your OpenCog work fails, so you don't kill everybody!!!
(One amusing/alarming quote from a commentator (probably not someone directly affiliated with SIAI) was "if you go ahead with an AGI when you're not 100% sure that it's safe, you're committing the Holocaust." But it wasn't just one extreme commentator, it was a bunch … and then a bunch of others commenting to me privately via email.)
If one fully accepts SIAI's Scary Idea, then one should not work on practical AGI projects, nor should one publish papers on the theory of how to build AGI systems. Instead, one should spend one's time trying to figure out an AGI design that is somehow provable-in-advance to be a Good Guy. For this reason, SIAI's research group is not currently trying to do any practical AGI work.
Actually, so far as I know, my "GOLEM" AGI design (mentioned above) is closer to a "provably Friendly AI" than anything the SIAI research team has come up with. At least, it's closer than anything they have made public.
However GOLEM is not something that could be practically implemented in the near future. It's horribly computationally inefficient, compared to a real-world AGI design like the OpenCog system I'm now working on (with many others -- actually I'm doing very little programming these days, so happily the project is moving forward with the help of others on the software design and coding side, while I contribute at the algorithm, math, design, theory, management and fundraising levels).
I agree that AGI ethics is a Very Important Problem. But I doubt the problem is most effectively addressed by theory alone. I think the way to come to a useful real-world understanding of AGI ethics is going to be to
- build some early-stage AGI systems, e.g. artificial toddlers, scientists' helpers, video game characters, robot maids and butlers, etc.
- study these early-stage AGI systems empirically, with a focus on their ethics as well as their cognition
- in the usual manner of science, attempt to arrive at a solid theory of AGI intelligence and ethics based on a combination of conceptual and experimental-data considerations
- humanity collectively plots the next steps from there, based on the theory we find: maybe we go ahead and create a superhuman AI capable of hard takeoff, maybe we pause AGI development because of the risks, maybe we build an "AGI Nanny" to watch over the human race and prevent AGI or other technologies from going awry. Whatever choice we make then, it will be made based on far better knowledge than we have right now.
Nothing, really -- if you hold the views of most AI researchers or futurists. There are plenty of disagreements about the right path to AGI, but wide and implicit agreement that something like the above path is sensible.
But, if you adhere to SIAI's Scary Idea, there's a big problem with this approach -- because, according to the Scary Idea, there's too huge of a risk that these early-stage AGI systems are going to experience a hard takeoff and self-modify into something that will destroy us all.
But I just don't buy the Scary Idea.
I do see a real risk that, if we proceed in the manner I'm advocating, some nasty people will take the early-stage AGIs and either use them for bad ends, or proceed to hastily create a superhuman AGI that then does bad things of its own volition. These are real risks that must be thought about hard, and protected against as necessary. But they are different from the Scary Idea. And they are not so different from the risks implicit in a host of other advanced technologies.
Conclusion
So, there we go.
I think SIAI is performing a useful service by helping bring these sorts of ideas to the attention of the futurist community (alongside the other services they're performing, like the wonderful Singularity Summits). But, that said, I think the Scary Idea is potentially a harmful one. At least, it WOULD be a harmful one, if more people believed it; so I'm glad it's currently restricted to a rather small subset of the futurist community.
Many people die each day, and many others are miserable for various reasons -- and all sorts of other advanced and potentially dangerous technologies are currently under active development. My own view is that unaided human minds may well be unable to deal with the complexity and risk of the world that human technology is unleashing. I actually suspect that our best hope for survival and growth through the 21st century is to create advanced AGIs to help us on our way -- to cure disease, to develop nanotech and better AGI and invent new technologies; and to help us keep nasty people from doing destructive things with advanced technology.
I think that to avoid actively developing AGI, out of speculative concerns like the Scary Idea, would be an extremely bad idea.
That is, rather than "if you go ahead with an AGI when you're not 100% sure that it's safe, you're committing the Holocaust," I suppose my view is closer to "if you avoid creating beneficial AGI because of speculative concerns, then you're killing my grandma" !! (Because advanced AGI will surely be able to help us cure human diseases and vastly extend and improve human life.)
So perhaps I could adopt the slogan: "You don't have to kill my grandma to avoid the Holocaust!" … but really, folks… Well, you get the point….
Humanity is on a risky course altogether, but no matter what I decide to do with my life and career (and no matter what Bill Joy or Jaron Lanier or Bill McKibben, etc., write), the race is not going to voluntarily halt technological progress. It's just not happening.
We just need to accept the risk, embrace the thrill of the amazing time we were born into, and try our best to develop near-inevitable technologies like AGI in a responsible and ethical way.
And to me, responsible AGI development doesn't mean fixating on speculative possible dangers and halting development until ill-defined, likely-unsolvable theoretical/philosophical issues are worked out to everybody's (or some elite group's) satisfaction.
Rather, it means proceeding with the work carefully and openly, learning what we can as we move along -- and letting experiment and theory grow together ... as they have been doing quite successfully for the last few centuries, at a fantastically accelerating pace.
And so it goes.
Wednesday, October 13, 2010
Let's Turn Nauru Into Transtopia
Here's an off-the-wall idea that has some appeal to me ... as a long-time Transtopian fantasist and world traveler....
The desert island nation of Nauru needs money badly, and has a population of less than 15,000
There are problems with water supply, but they could surely be solved with some technical ingenuity.
The land area is about 8 square miles. But it could be expanded! Surely it's easier to extend an island with concrete platforms or anchored floating platforms of some other kind, than to seastead in the open ocean.
The country is a democracy. Currently it may not be possible to immigrate there except as a temporary tourist or business visitor. But I'd bet this could be made negotiable.
Suppose 15,000 adult transhumanists (along with some kids, one would assume) decided to emigrate to Nauru en masse over a 5-year period, on condition they could obtain full citizenship. Perhaps this could be negotiated with the Nauruan government.
Then after 5 years we would have a democracy in which transhumanists were the majority.
Isn't this the easiest way to create a transhumanist nation? With all the amazing future possibilities that that implies?
This would genuinely be of benefit to the residents of Nauru, which now has 90% unemployment. Unemployment would be reduced close to zero, and the economy would be tremendously enlarged. A win-win situation. Transhumanists would get freedom, and Nauruans would get a first-world economy.
Considerable infrastructure would need to be built. A deal would need to be struck with the government, in which, roughly,
This much emigration would make Nauru crowded, but not nearly as crowded as some cities. And with a seasteading mindset, it's easy to see that the island is expandable.
To ensure employment of the relocated transhumanists, we would need to get a number of companies to agree to open Nauru offices. But this would likely be tractable, given the preference of firms to have offices in major tech centers. Living expenses in Nauru would be much lower than in, say, Silicon Valley, so expenses would be lower.
Tourism could become a major income stream, given the high density of interesting people which would make Nauru into a cultural mecca. Currently there is only one small beach on Nauru (which is said to be somewhat dirty), but creation of a beautiful artificial beach on the real ocean is not a huge technological feat.
It would also be a great place to experiment with aquaculture and vertical farming.
What say you? Let's do it!
P.S.
Other candidates for the tropical island Transtopia besides Nauru would be Tuvalu and Kiribati; but Kiribati's population is much larger, and Tuvalu is spread among many islands, and is also about to become underwater due to global warming. So Nauru would seem the number one option. Though, Tuvalu could be an interesting possibility also, especially if we offered to keep the island above water by building concrete platforms or some such (a big undertaking, but much easier than seasteading). This would obviously be a major selling point to the government.
The desert island nation of Nauru needs money badly, and has a population of less than 15,000
There are problems with water supply, but they could surely be solved with some technical ingenuity.
The land area is about 8 square miles. But it could be expanded! Surely it's easier to extend an island with concrete platforms or anchored floating platforms of some other kind, than to seastead in the open ocean.
The country is a democracy. Currently it may not be possible to immigrate there except as a temporary tourist or business visitor. But I'd bet this could be made negotiable.
Suppose 15,000 adult transhumanists (along with some kids, one would assume) decided to emigrate to Nauru en masse over a 5-year period, on condition they could obtain full citizenship. Perhaps this could be negotiated with the Nauruan government.
Then after 5 years we would have a democracy in which transhumanists were the majority.
Isn't this the easiest way to create a transhumanist nation? With all the amazing future possibilities that that implies?
This would genuinely be of benefit to the residents of Nauru, which now has 90% unemployment. Unemployment would be reduced close to zero, and the economy would be tremendously enlarged. A win-win situation. Transhumanists would get freedom, and Nauruans would get a first-world economy.
Considerable infrastructure would need to be built. A deal would need to be struck with the government, in which, roughly,
- They agreed to allow a certain number of outsiders citizenship, and to allow certain infrastructure development
- Over a couple years, suitable infrastructure was built to supply electrical power, Internet, more frequent flights, etc.
- Then, over a few years after that, the new population would flow in
This much emigration would make Nauru crowded, but not nearly as crowded as some cities. And with a seasteading mindset, it's easy to see that the island is expandable.
To ensure employment of the relocated transhumanists, we would need to get a number of companies to agree to open Nauru offices. But this would likely be tractable, given the preference of firms to have offices in major tech centers. Living expenses in Nauru would be much lower than in, say, Silicon Valley, so expenses would be lower.
Tourism could become a major income stream, given the high density of interesting people which would make Nauru into a cultural mecca. Currently there is only one small beach on Nauru (which is said to be somewhat dirty), but creation of a beautiful artificial beach on the real ocean is not a huge technological feat.
It would also be a great place to experiment with aquaculture and vertical farming.
What say you? Let's do it!
P.S.
Other candidates for the tropical island Transtopia besides Nauru would be Tuvalu and Kiribati; but Kiribati's population is much larger, and Tuvalu is spread among many islands, and is also about to become underwater due to global warming. So Nauru would seem the number one option. Though, Tuvalu could be an interesting possibility also, especially if we offered to keep the island above water by building concrete platforms or some such (a big undertaking, but much easier than seasteading). This would obviously be a major selling point to the government.
Sunday, October 10, 2010
What Would It Take to Move Rapidly Toward Beneficial Human-Level AGI?
On Thursday I finished writing the last chapter of my (co-authored) two-volume book on how to create beneficial human-level AGI, Building Better Minds. I have a bunch of editing still do so, some references to add, etc. -- but the book is now basically done. Woo hoo!
The book should be published by a major scientific publisher sometime in 2011.
The last chapter describes, in moderate detail, how the CogPrime cognitive architecture (implemented in the OpenCog open-source framework) would enable a robotic or virtual embodied system to appropriately respond to the instruction "Build me something surprising out of blocks." This is in the spirit of the overall idea: Build an AGI toddler first, then teach it, study it, and use it as a platform to go further.
From an AGI toddler, I believe, one could go forward in a number of directions: toward fairly human-like AGIs, but also toward different sorts of minds formed by hybridizing the toddler with narrow-AI systems carrying out particular classes of tasks in dramatically transhuman ways.
Reading through the 900-page tome my colleagues and I have put together, I can't help reflecting on how much work is left to bring it all into reality! We have a software framework that is capable of supporting the project (OpenCog), and we have a team of people capable of doing it (people working with me on OpenCog now; people working with me on other projects now; people I used to work with but who moved on to other things, but would enthusiastically come back for a well-funded AGI project). We have a rich ecosystem of others (e.g. academic and industry AI researchers, as well as neuroscientists, philosophers, technologists, etc. etc.) who are enthusiastic to provide detailed, thoughtful advice as we proceed.
What we don't have is proper funding to implement the stuff in the book and create the virtual toddler!
This is of course a bit frustrating: I sincerely believe I have a recipe for creating a human-level thinking machine! In an ethical way, and with computing resources currently at our disposal.
But implementing this recipe would be a lot of work, involving a number of people working together in a concentrated and coordinated way over a significant period of time.
I realize I could be wrong, or I could be deluding myself. But I've become a lot more self-aware and a lot more rational through my years of adult life (I'm 43 now), and I really don't think so. I've certainly introspected and self-analyzed a lot to understand the extent to which I may be engaged in wishful thinking about AGI, and my overall conclusion (in brief) is as follows: Estimating timing is hard, for any software project, let alone one involving difficult research. And there are multiple PhD-thesis-level research problems that need to be solved in the midst of getting the whole CogPrime design to work (but by this point in my career, I believe I have a decent intuition for distinguishing tractable PhD-thesis-level research problems from intractable conundrums). And there's always the possibility of the universe being way, way different than any of us understands, in some way that stops any AGI design based on digital computers (or any current science!) from working. But all in all, evaluated objectively according to my professional knowledge, the whole CogPrime design appears sensible -- if all the parts work vaguely as expected, the whole system should lead to human-level AGI; and according to current computer science and narrow AI theory and practice, all the parts are very likely to work roughly as expected.
So: I have enough humility and breadth to realize I could be wrong, but I have studied pretty much all the relevant knowledge that's available, I've thought about this hard for a very long time and talked to a large percentage of the world's (other) experts; I'm not a fool and I'm not self-deluded in some shallow and obvious way. And I really believe this design can work!
It's the same design I've been refining since about 1996. The prototyping my colleagues and I did at Webmind Inc. (when we had a 45-person AGI research team) in 1998-2001 was valuable, both for what it taught us about what NOT to do and for positive lessons. The implementation work my colleagues at Novamente LLC and the OpenCog project have done since 2001 has been very valuable too; and it's led to an implementation of maybe 40% of the CogPrime design (depending on how you measure it). (But unfortunately 40% of a brain doesn't yield 40% of the functionality of a whole brain, particularly because in this case (beyond the core infrastructure) the 40% implemented has been largely chosen by what was useful for Novamente LLC application projects rather than what we thought would serve best as the platform for AGI.) Having so many years to think through the design, without a large implementation team to manage, has been frustrating but also good in a sense, in that it's given me and my colleagues time and space to repeatedly mull over the design and optimize it in various ways.
Now, the funding situation for the project is not totally dismal, or it least it doesn't seem so right now. For that I am grateful.
The OpenCog project does appear to be funded, at least minimally, for the next couple years. This isn't quite 100% certain, but it's close -- it seems we've lined up funding for a handful of people to work full-time on a fairly AGI-ish OpenCog application for 2 years (I'll post here about this at length once it's definite). And there's also the Xiamen University "Brain-Like Intelligent Systems" lab, in which some grad students are applying OpenCog to enable some intelligent robotic behaviors. And Novamente LLC is still able to fund a small amount of OpenCog work, via application projects that entail making some improvements to the OpenCog infrastructure along the way. So all in all, it seems, we'll probably continue making progress, which is great.
But I'm often asked, by various AGI enthusiasts, what it would take to make really fast progress toward my AGI research goals. What kind of set-up, what kind of money? Would it take a full-on "AGI Manhattan Project" -- or something smaller?
In the rest of this blog post I'm going to spell it out. The answer hasn't changed much for the last 5 years, and most likely won't change a lot during the next 5 (though I can't guarantee that).
What I'm going to describe is the minimal team required to make reasonably fast progress. Probably we could progress even faster if we had massively more funding, but I'm trying to be realistic here.
We could use a team of around 10 of the right people (mostly, great AI programmers, with a combination of theory understanding and implementation chops), working full-time on AI development.
We could use around 5 great programmers working on the infrastructure -- to get OpenCog working really efficiently on a network of distributed multi-processor machines.
If we're going to do robotics, we could use a dedicated robotics team of perhaps 5 people.
If we're going to do virtual agents, we could use 5 people working on building out the virtual world appropriately for AGI.
Add a system administrator, 2 software testers, a project manager to help us keep track of everything, and a Minister of Information to help us keep all the documentation in order.
That's 30 people. Then add me and my long-time partner Cassio Pennachin to coordinate the whole thing (and contribute to the technical work as needed), and a business manager to help with money and deal with the outside world. 33 people.
Now let's assume this is done in the US (not the only possibility, but the simplest one to consider), and let's assume we pay people close to market salaries and benefits, so that their spouses don't get mad at them and decrease their productivity (yes, it's really not optimal to do a project like this with programmers fresh out of college -- this isn't a Web 2.0 startup, it's a massively complex distributed software system based on integration of multiple research disciplines. Many of the people with the needed expertise have spouses, families, homes, etc. that are important to them). Let's assume it's not done in Silicon Valley or somewhere else where salaries are inflated, but in some other city with a reasonable tech infrastructure and lower housing costs. Then maybe, including all overheads, we're talking about $130K/year per employee (recall that we're trying to hire the best people here; some are very experienced and some just a few years out of college, but this is an average).
Salary cost comes out to $4.3M/year, at this rate.
Adding in a powerful arsenal of hardware and a nice office, we can round up to $5M/year
Let's assume the project runs for 5 years. My bet is we can get an AGI toddler by that time. But even if that's wrong, I'm damn sure we could make amazing progress by that time, suitable to convince a large number of possible funding sources to continue funding the project at the same or a greater level.
Maybe we can do it in 3 years, maybe it would take 7-8 years to get to the AGI toddler goal -- but even if it's the latter, we'd have amazing, clearly observable dramatic progress in 3-5 years.
So, $25M total.
There you go. That's what it would cost to progress toward human-level AGI, using the CogPrime design, in a no-BS straightforward way -- without any fat in the project, but also without cutting corners in ways that reduce efficiency.
If we relax the assumption that the work is done in the US and move to a less expensive place (say, Brazil or China where OpenCog already has some people working) we can probably cut thc cost by half without a big problem. We would lose some staff who wouldn't leave the US, so there would be a modest decrease in productivity, but it wouldn't kill the project. (Why does it only cut the cost by half? Because if we're importing first-worlders to the Third World to save money, we still need to pay them enough to cover expenses they may have back in the US, to fly home to see their families, etc.)
So, outside the US, $13M total over 5 years.
Or if we want to rely more on non-US people for some of the roles (e.g. systems programming, virtual worlds,...), it can probably be reduced to $10M total over 5 years, $2M/year.
If some wealthy individual or institution were willing to put in $10M -- or $25M if they're fixated on a US location (or, say, $35M if they're fixated on Silicon Valley) -- then we could progress basically full-speed-ahead toward creating beneficial human-level AGI.
Instead, we're progressing toward the same goal seriously and persistently, but much more slowly and erratically.
I have spoken personally to a decent number of individuals with this kind of money at their disposal, and many of them are respectful of and interested in the OpenCog project -- and would be willing to put in this kind of money if they had sufficient confidence the project would succeed.
But how to give potential funders this sort of confidence?
After all, when they go to the AI expert at their local university, the guy is more likely than not to tell them that human-level AI is centuries off. Or if they open up The Singularity is Near, by Ray Kurzweil who is often considered a radical techno-optimist, they see a date of 2029 for human-level AGI -- which means that as investors they would probably start worrying about it around 2025.
A 900-page book is too much to expect a potential donor or investor to read; and even if they read it (once its published), it doesn't give an iron-clad irrefutable argument that the project will succeed, "just" a careful overall qualitative argument together with detailed formal treatments of various components of the design.
The various brief conference papers I've published on the CogPrime design and OpenCog project, give a sense of the overall spirit but don't tell you enough to let you make a serious evaluation. Maybe this is a deficiency in the writing, but I suspect it's mainly a consequence of the nature of the subject matter.
The tentative conclusion that I've come to is that, barring some happy luck, we will need to come up with some amazing demo of AGI functionality -- something that will serve as an "AGI Sputnik" moment.
Sputnik, of course, caused the world to take space flight seriously. The right AGI demo could do the same. It could get OpenCog funded as described above, plus a lot of other AGI projects in parallel.
But the question is, how to get to the AGI Sputnik moment without the serious funding. A familiar, obvious chicken-and-egg problem.
One possibility is to push far enough toward a virtual toddler in a virtual world, using our current combination of very-much-valued but clearly-suboptimal funding sources -- that our animated AGI baby has AGI Sputnik power!
Maybe this will happen. I'm certainly willing to put my heart into it, and so are a number of my colleagues.
But it sure is frustrating to know that, for an amount of money that's essentially "pocket change" to a significant number of individuals and institutions on the planet, we could be progressing a lot faster toward some goals that are really important to all of us.
To quote Kurt Vonnegut: "And so it goes."
The book should be published by a major scientific publisher sometime in 2011.
The last chapter describes, in moderate detail, how the CogPrime cognitive architecture (implemented in the OpenCog open-source framework) would enable a robotic or virtual embodied system to appropriately respond to the instruction "Build me something surprising out of blocks." This is in the spirit of the overall idea: Build an AGI toddler first, then teach it, study it, and use it as a platform to go further.
From an AGI toddler, I believe, one could go forward in a number of directions: toward fairly human-like AGIs, but also toward different sorts of minds formed by hybridizing the toddler with narrow-AI systems carrying out particular classes of tasks in dramatically transhuman ways.
Reading through the 900-page tome my colleagues and I have put together, I can't help reflecting on how much work is left to bring it all into reality! We have a software framework that is capable of supporting the project (OpenCog), and we have a team of people capable of doing it (people working with me on OpenCog now; people working with me on other projects now; people I used to work with but who moved on to other things, but would enthusiastically come back for a well-funded AGI project). We have a rich ecosystem of others (e.g. academic and industry AI researchers, as well as neuroscientists, philosophers, technologists, etc. etc.) who are enthusiastic to provide detailed, thoughtful advice as we proceed.
What we don't have is proper funding to implement the stuff in the book and create the virtual toddler!
This is of course a bit frustrating: I sincerely believe I have a recipe for creating a human-level thinking machine! In an ethical way, and with computing resources currently at our disposal.
But implementing this recipe would be a lot of work, involving a number of people working together in a concentrated and coordinated way over a significant period of time.
I realize I could be wrong, or I could be deluding myself. But I've become a lot more self-aware and a lot more rational through my years of adult life (I'm 43 now), and I really don't think so. I've certainly introspected and self-analyzed a lot to understand the extent to which I may be engaged in wishful thinking about AGI, and my overall conclusion (in brief) is as follows: Estimating timing is hard, for any software project, let alone one involving difficult research. And there are multiple PhD-thesis-level research problems that need to be solved in the midst of getting the whole CogPrime design to work (but by this point in my career, I believe I have a decent intuition for distinguishing tractable PhD-thesis-level research problems from intractable conundrums). And there's always the possibility of the universe being way, way different than any of us understands, in some way that stops any AGI design based on digital computers (or any current science!) from working. But all in all, evaluated objectively according to my professional knowledge, the whole CogPrime design appears sensible -- if all the parts work vaguely as expected, the whole system should lead to human-level AGI; and according to current computer science and narrow AI theory and practice, all the parts are very likely to work roughly as expected.
So: I have enough humility and breadth to realize I could be wrong, but I have studied pretty much all the relevant knowledge that's available, I've thought about this hard for a very long time and talked to a large percentage of the world's (other) experts; I'm not a fool and I'm not self-deluded in some shallow and obvious way. And I really believe this design can work!
It's the same design I've been refining since about 1996. The prototyping my colleagues and I did at Webmind Inc. (when we had a 45-person AGI research team) in 1998-2001 was valuable, both for what it taught us about what NOT to do and for positive lessons. The implementation work my colleagues at Novamente LLC and the OpenCog project have done since 2001 has been very valuable too; and it's led to an implementation of maybe 40% of the CogPrime design (depending on how you measure it). (But unfortunately 40% of a brain doesn't yield 40% of the functionality of a whole brain, particularly because in this case (beyond the core infrastructure) the 40% implemented has been largely chosen by what was useful for Novamente LLC application projects rather than what we thought would serve best as the platform for AGI.) Having so many years to think through the design, without a large implementation team to manage, has been frustrating but also good in a sense, in that it's given me and my colleagues time and space to repeatedly mull over the design and optimize it in various ways.
Now, the funding situation for the project is not totally dismal, or it least it doesn't seem so right now. For that I am grateful.
The OpenCog project does appear to be funded, at least minimally, for the next couple years. This isn't quite 100% certain, but it's close -- it seems we've lined up funding for a handful of people to work full-time on a fairly AGI-ish OpenCog application for 2 years (I'll post here about this at length once it's definite). And there's also the Xiamen University "Brain-Like Intelligent Systems" lab, in which some grad students are applying OpenCog to enable some intelligent robotic behaviors. And Novamente LLC is still able to fund a small amount of OpenCog work, via application projects that entail making some improvements to the OpenCog infrastructure along the way. So all in all, it seems, we'll probably continue making progress, which is great.
But I'm often asked, by various AGI enthusiasts, what it would take to make really fast progress toward my AGI research goals. What kind of set-up, what kind of money? Would it take a full-on "AGI Manhattan Project" -- or something smaller?
In the rest of this blog post I'm going to spell it out. The answer hasn't changed much for the last 5 years, and most likely won't change a lot during the next 5 (though I can't guarantee that).
What I'm going to describe is the minimal team required to make reasonably fast progress. Probably we could progress even faster if we had massively more funding, but I'm trying to be realistic here.
We could use a team of around 10 of the right people (mostly, great AI programmers, with a combination of theory understanding and implementation chops), working full-time on AI development.
We could use around 5 great programmers working on the infrastructure -- to get OpenCog working really efficiently on a network of distributed multi-processor machines.
If we're going to do robotics, we could use a dedicated robotics team of perhaps 5 people.
If we're going to do virtual agents, we could use 5 people working on building out the virtual world appropriately for AGI.
Add a system administrator, 2 software testers, a project manager to help us keep track of everything, and a Minister of Information to help us keep all the documentation in order.
That's 30 people. Then add me and my long-time partner Cassio Pennachin to coordinate the whole thing (and contribute to the technical work as needed), and a business manager to help with money and deal with the outside world. 33 people.
Now let's assume this is done in the US (not the only possibility, but the simplest one to consider), and let's assume we pay people close to market salaries and benefits, so that their spouses don't get mad at them and decrease their productivity (yes, it's really not optimal to do a project like this with programmers fresh out of college -- this isn't a Web 2.0 startup, it's a massively complex distributed software system based on integration of multiple research disciplines. Many of the people with the needed expertise have spouses, families, homes, etc. that are important to them). Let's assume it's not done in Silicon Valley or somewhere else where salaries are inflated, but in some other city with a reasonable tech infrastructure and lower housing costs. Then maybe, including all overheads, we're talking about $130K/year per employee (recall that we're trying to hire the best people here; some are very experienced and some just a few years out of college, but this is an average).
Salary cost comes out to $4.3M/year, at this rate.
Adding in a powerful arsenal of hardware and a nice office, we can round up to $5M/year
Let's assume the project runs for 5 years. My bet is we can get an AGI toddler by that time. But even if that's wrong, I'm damn sure we could make amazing progress by that time, suitable to convince a large number of possible funding sources to continue funding the project at the same or a greater level.
Maybe we can do it in 3 years, maybe it would take 7-8 years to get to the AGI toddler goal -- but even if it's the latter, we'd have amazing, clearly observable dramatic progress in 3-5 years.
So, $25M total.
There you go. That's what it would cost to progress toward human-level AGI, using the CogPrime design, in a no-BS straightforward way -- without any fat in the project, but also without cutting corners in ways that reduce efficiency.
If we relax the assumption that the work is done in the US and move to a less expensive place (say, Brazil or China where OpenCog already has some people working) we can probably cut thc cost by half without a big problem. We would lose some staff who wouldn't leave the US, so there would be a modest decrease in productivity, but it wouldn't kill the project. (Why does it only cut the cost by half? Because if we're importing first-worlders to the Third World to save money, we still need to pay them enough to cover expenses they may have back in the US, to fly home to see their families, etc.)
So, outside the US, $13M total over 5 years.
Or if we want to rely more on non-US people for some of the roles (e.g. systems programming, virtual worlds,...), it can probably be reduced to $10M total over 5 years, $2M/year.
If some wealthy individual or institution were willing to put in $10M -- or $25M if they're fixated on a US location (or, say, $35M if they're fixated on Silicon Valley) -- then we could progress basically full-speed-ahead toward creating beneficial human-level AGI.
Instead, we're progressing toward the same goal seriously and persistently, but much more slowly and erratically.
I have spoken personally to a decent number of individuals with this kind of money at their disposal, and many of them are respectful of and interested in the OpenCog project -- and would be willing to put in this kind of money if they had sufficient confidence the project would succeed.
But how to give potential funders this sort of confidence?
After all, when they go to the AI expert at their local university, the guy is more likely than not to tell them that human-level AI is centuries off. Or if they open up The Singularity is Near, by Ray Kurzweil who is often considered a radical techno-optimist, they see a date of 2029 for human-level AGI -- which means that as investors they would probably start worrying about it around 2025.
A 900-page book is too much to expect a potential donor or investor to read; and even if they read it (once its published), it doesn't give an iron-clad irrefutable argument that the project will succeed, "just" a careful overall qualitative argument together with detailed formal treatments of various components of the design.
The various brief conference papers I've published on the CogPrime design and OpenCog project, give a sense of the overall spirit but don't tell you enough to let you make a serious evaluation. Maybe this is a deficiency in the writing, but I suspect it's mainly a consequence of the nature of the subject matter.
The tentative conclusion that I've come to is that, barring some happy luck, we will need to come up with some amazing demo of AGI functionality -- something that will serve as an "AGI Sputnik" moment.
Sputnik, of course, caused the world to take space flight seriously. The right AGI demo could do the same. It could get OpenCog funded as described above, plus a lot of other AGI projects in parallel.
But the question is, how to get to the AGI Sputnik moment without the serious funding. A familiar, obvious chicken-and-egg problem.
One possibility is to push far enough toward a virtual toddler in a virtual world, using our current combination of very-much-valued but clearly-suboptimal funding sources -- that our animated AGI baby has AGI Sputnik power!
Maybe this will happen. I'm certainly willing to put my heart into it, and so are a number of my colleagues.
But it sure is frustrating to know that, for an amount of money that's essentially "pocket change" to a significant number of individuals and institutions on the planet, we could be progressing a lot faster toward some goals that are really important to all of us.
To quote Kurt Vonnegut: "And so it goes."
Tuesday, September 28, 2010
Mind Uploading via Gmail
Cut and pasted from Giulio Prisco's blog here
(with one small change)
...
Mind Uploading via GmailI am writing this in 2010. My Gmail account has more than 20GB of data, which contain some information about me and also some information about the persons I have exchanged email with, including some personal and private information.
I am assuming that in 2060 (50 years from now), my Gmail account will have hundreds or thousands of TB of data, which will contain a lot of information about me and the persons I exchanged email with, including a lot of personal and private information. I am also assuming that, in 2060:
- The data in the accounts of all Gmail users since 2004 is available.
- AI-based mindware technology able to reconstruct individual mindfiles by analyzing the information in their aggregate Gmail accounts and other available information, with sufficient accuracy for mind uploading via detailed personality reconstruction, is available.
- The technology to crack Gmail passwords is available, but illegal without the consent of the account owners (or their heirs).
- Many of today's Gmail users, including myself, are already dead and cannot give permission to use the data in their accounts.
If all assumptions above are correct, I hereby give permission to Google and/or other parties to read all data in my Gmail account and use them together with other available information to reconstruct my mindfile with sufficient accuracy for mind uploading via detailed personality reconstruction, and express my wish that they do so.
Signed by Ben Goertzel on September 28, 2010, and witnessed by readers.
NOTE: The accuracy of the process outlined above increases with the number of persons who give their permission to do the same. You can give your permission in comments, Twitter or other public spaces.
Sunday, August 08, 2010
RIP Lev Goertzel Mann, 1995-2010
The last "obituary" blog post I wrote was for my grandfather Leo Zwell -- the man who taught me about science and so much else. He died at age 91, after a long life rich in personal, professional and intellectual satisfaction. His death was tragic, as are almost all deaths. But the death I'm noting in this (painfully inadequate) post is vastly more depressing and tragic.
My sister's son Lev died last month, just short of his 15'th birthday. His death was totally unanticipated -- he was on vacation with his parents and his brother Jaal, camping in the forest in Alberta (Canada), and was struck in the night with a bizarre sudden illness. He stopped breathing a minute from the hospital, and died there an hour and a half later. The cause of death remains unclear, but the autopsy revealed a severe brain infection of some sort. One serious possibility is some form of meningitis (there are forms with quite brief incubation periods, i.e. a handful of days).
Think about it: One day you're totally healthy … then the next day, you die in your sleep, never having known anything was wrong. Maybe a mild headache the day before, no different than dozens or hundreds of other headaches you've had in your life.
My friend and colleague Jeff Pressing died in a very similar way a decade ago, in the midst of his adulthood.
I visited my sister and her family for 4 days shortly after Lev's death (they live near Seattle, I live in Maryland), but even so, I can't imagine what they're going through.
A full obituary is here.
My extended family is fairly small, but even so, there are some relatives I'm closer to than others. Lev was among the closer ones. He always felt like part of the same cognitive/spiritual tribe as me and my own kids: weird, sarcastic, outrageous sometimes, difficult and obnoxious sometimes, intellectual, adventurous, curious, warm-hearted, courageous, playful, compassionate, constantly skeptical but willing to steadfastly defend his best-guess beliefs and his sometimes-odd tastes.
Like me, Lev was called "crazy" many times, but usually in an affectionate way -- but of course he wasn't really crazy, just a free thinker unwilling to take anyone's word for anything or take anything for granted, dedicated to find his own path to enjoyment and understanding.
Though we never lived nearby each other, we saw each other at least once a year, sometimes more. Most recently he had spent a week at my house in April 2010 -- 3 months before his death. It was a great visit, with some fascinating conversations as well as lots of video games, frisbee, hiking, rock-climbing and so forth. I was struck by how fast he was growing up all of a sudden. Lev had always been a smart and inquisitive kid, but on this visit he was more interested to carry out lengthy intellectual chats -- about DNA, time travel, AI and so forth. He also showed a deep knowledge of history and politics, with an insight into Western history complementing my own sons' recent study of Japanese and Mongolian history. We even discussed the possibility of immortality via technological means, and he was all in favor.
He was a devout heavy metal head, and particularly a devotee of Metallica. I failed to convert him to jazz fusion, though he admitted that some of it sounded a bit like music. Like many teenagers, he had mused on death frequently, and long previously had told his parents the song he wanted played at his funeral, if he were ever to die: Fade to Black, by Metallica.
Fade to Black was indeed played at the funeral, which was the point in the funeral where I finally "lost it" and cried in a way I hadn't for a very long time. I played that song many times in the week following. Though I prefer Master of Puppets as a piece of music, obviously his choice was highly apropos for the setting. Yet the lyrics didn't quite fit. The lyrics say
Life it seems will fade away
Drifting further every day
Getting lost within myself
Nothing matters, no one else
I have lost the will to live
Simply nothing more to give
There is nothing more for me
Need the end to set me free
Things not what they used to be
Missing one inside of me
Deathly lost, this can't be real
Cannot stand this hell I feel
Emptiness is filling me
To the point of agony
Growing darkness taking dawn
I was me but now he's gone
No one but me can save myself
But it's too late
Now I can't think
Think why I should even try
Yesterday seems as though
It never existed
Death greets me warm
Now I will just say goodbye
but certainly Lev had NOT lost the will to live, except perhaps in the last few hours when he was unconscious and his body was succumbing to the infection that killed him. He was full of enthusiasm for life and excitement for the future.
"I was me but now he's gone."
He and his best friend Zay had plans to go to university together in Switzerland (I don't know why they chose that country). They had been best friends many years earlier when they went to school together in Costa Rica, and had maintained their friendship over many years via 7+ hours per week of phone calls -- wide-ranging phone calls, sometimes occupied with an expansive, multi-year collaboratively-created "imaginary adventure game"; sometimes with conversation on serious or casual topics; sometimes with long pauses while one or another worked on homework while the phone line was kept open.
The funeral was Quaker style, meaning that there was no primary speaker, but rather the individuals in the audience were invited to stand up and state their memories of Lev. There were many moving speeches but to me the most touching and insightful was Zay's. Zay recounted Tegmark's variant of multiverse theory, according to which -- due to the large extent and general quasi-randomness of the universe -- it's likely that the universe contains multiple variants of Earth, each of which is similar to our own but with minor variations. Zay pointed out that, in this case, there would be many variant Earths, including many on which Lev did not get an infection and die at age 14. He said he took some small solace from the fact that, in those variant Earths, his analogue and Lev's analogue would get to grow up together and experience adulthood together.
At my sister's house, after the funeral, we each took some of Lev's things, to symbolize his memory. My 13 year old daughter Scheherazade took one of the stories he'd hand-written in a small notebook. On the front page, alongside the title, was scribbled the following marginal note: "Kick Zay in the testicles." I was reminded of a time, years ago, when I invited the 9 year old Lev to "play fight" with me, in the manner one often does with young children. He immediately initiated the fight by kicking me in the nuts as hard as he could. No particular hostility was intended -- he was just play-fighting, Lev-style. A few minutes later he wandered off in some random direction, lost in imagination, and we had to hunt him down -- a common habit when in public with Lev, especially when he was younger.
As I pointed out to Zay in a conversation after the funeral, if Tegmark's theory is correct, then it's possible some future technology could allow us to visit those variant Earths one day, so he might actually get to see the 15, 25 and 85 year old Lev after all.
It's also possible that, as Martine Rothblatt, Bill Bainbridge and some other futurists speculate (see e.g. CyberEv), we may eventually be able to reconstitute deceased humans from data such as their writings, and recordings of their voice and physical appearance and movements, and their imprints on the memories of others.
But while I take such future possibilities seriously, they don't really help mute the tragedy much. Right now, in the world we know concretely, Lev is gone -- and I can't shake the feeling he shouldn't be.
There's some room for philosophical debate about the merits of death via old age. Some say death is natural, and therefore aesthetically and morally positive. Some say it lends a particular meaningfulness and elegance to life, and that without it life would lose depth and pizazz. I don't really buy that. Of course death adds meaning to life, and of course there is a certain aesthetic charm to a life that ends in death, which wouldn't be there in an infinite life. But an infinite life would have a different kind of depth and pizazz -- and probably ultimately a much better kind. There is also a special meaningfulness and elegance to being tortured, or dying of cancer, but yet we don't crave these, we try to avoid them -- because we prefer other forms of meaningfulness and elegance.
But anyway, all that is moot in this case -- only the most cognitively-distorted religious fanatic would argue the merits of the sudden death by disease of a healthy, vibrant child.
When my grandfather Leo died at age 91, it was terribly sad, but there was a certain feeling of completeness to the story. He had anticipated his death for a while -- he had no expectations of afterlife, and he had reconciled himself to his own impending nonexistence. Ever a scientist, he came to see his limited time-scope as being roughly comparable to his limited space-scope. He'd had a long and rich marriage, children, grandchildren, great-grandchildren,…. His life had followed a meaningful arc. I certainly wish Leo had lived forever -- but, the wrongness of his death was pretty much restricted to the wrongness implicit in the general wrongness of the human condition.
But Lev dying at age 14 is another matter. It's like finding the bottom 1/5 of a Picasso painting, or an unfinished symphony with only the first of 7 movements -- and a tiny fragment of the second, cut off at a completely senseless place. There's no positive aesthetic or moral value in such a death. It just fucking sucks. There's no obscene invective, and no poetic prose nor obituary nor blog post, and no Metallica nor Beethoven song, capable to convey even fractionally the massive fucking-suckiness of such a thing.
Nietzche wrote about the merits of "dying at the right time." He felt a good death was just as important as a good life. Nietzsche himself egregiously failed to die at the right time, spending the last 11 years of his life mute and semi-insane from some sort of brain disease. Leo, perhaps, died at the right time (given the current, deeply flawed order of human existence). Lev massively did not.
I really hope to see Lev again on some variant Earth or in some computer simulation or other dimension or whatever. I have no way to confidently estimate the odds of such a thing. I do believe the world we now know and understand via science merely scratches the surface of the overall universe with its copious transhuman hidden patterns, orders and flows. But the mysteriousness of life doesn't imply that the universe will someday deliver us the various things we want.
For now, it's just really damn depressing.
And for all that, I still can't remotely imagine how I'd feel if it were one of my own kids.
I'll close with some pieces of advice from Lev, which were collected by his parents and emailed to friends and family and posted on the wall at the funeral:
- Color outside the lines and do it quickly.
- There is no need to be consistent.
- Be difficult. It is a winning strategy.
- Go outside in the pouring rain.
- Be intensely critical of everything all the time.
- There is nothing wrong with a good scowl. Practice if necessary.
- Spend your money quickly.
- If you love someone, give them your stuff.
- Go off-trail and climb to the top of the hill.
- Like yourself. You're awesome.
- Buy it now because you want it. Next week you won't care about it.
- When forced to eat vegetables, shove them all in your mouth at once, chew, and swallow. Then enjoy the rest of your meal.
- Love your friends; ignore your enemies.
- Don't listen to anything your parents say. They know way less than you do.
- Say whatever is on your mind...mean or nice.
- Wear a hat. Bare heads are boring.
- If what you are doing is not fun, it is not worth doing.
- You really can read 3 books at once.
- When everyone turns in the assignment, that's a good clue that you better get started on it.
- Live in the moment.
- Don't let anyone break a spirit.
- Speak out for what you believe in.
P.S.
This post-script is inspired by Lev's death rather than directly about Lev or his death. I'm sure Lev wouldn't mind.
After Lev's funeral I couldn't help imagining: what if the funeral were my own? What if I were the one struck randomly by some bizarre disease?
Of course, we always know intellectually that each day could be our last -- but we rarely live in a manner that richly incorporates this knowledge.
A few days after Lev's funeral I had a dream of my own funeral -- which was (in the dream) in the same building as Lev's, but of course with different music and different people. Also, it was winter outside, whereas Lev's occurred in summer.
Instead of "Fade to Black," my dream-funeral featured two songs in sequence: The Structure of Mind by me (the best song I've written); followed by Soothsayer, by Buckethead (the best song I wish I'd written).
Soothsayer, to me, is all about the presence of a hidden order, pattern and flow to the cosmos. It's about the presence of something else there in the world -- something bigger and wiser and crazier than us; some structured dynamic domain of being/becoming, which we can never quite understand without losing our pathetic little human selves. Buckethead's Soothsayer -- like Hendrix's Voodoo Child before him -- wants to lead us there, but we travel there at our peril. You have to choose: either retain your human form and forego the transcendent domain, except in dribs and drabs; or lose your self and open your heart and mind to the transcendent. Lev followed the Soothsayer. For now, still, here I am.
Also my daughter Scheherazade (who is now 13, but was older than that in the dream) read the following statement, which I had written before my death:
"
I'd like to thank my parents Carol and Ted for creating me and raising me. My grandfather Leo Zwell for teaching me about science. My kids Zar, Zeb and Zade for being awesome kids and giving a center to my life. My first wife Gwen and my second wife Izabela for all the good times and deep sharing. Gwen for giving me the kids as well. Cassio Pennachin for so much professional and intellectual partnership. Goodbye, and thanks for all the fish. Hope to see you all again in some other time, or some other dimension. As Jimi Hendrix said: If I don't see you no more in this world, I'll meet you in the next one, and don't be late.
"
I think there was more to the speech Zade read than that also, but I can't recall all the details.
Then Jimi Hendrix's Voodoo Chile (not Slight Return) was played, while people ate green eggs and ham (seriously!).
(Gwen and I made green eggs and ham for the kids once, back in the day, inspired by the Dr. Seuss book. The food coloring made the eggs taste funny.)
Voodoo Chile:
Well, the night I was born
Lord I swear the moon turned a fire red
The night I was born
I swear the moon turned a fire red
Well my poor mother cried out "lord, the gypsy was right!"
And I seen her fell down right dead
Well, mountain lions found me there waitin'
And set me on a eagles back
Well, mountain lions found me there,
And set me on a eagles wing
(Its' the eagles wing, baby, what did I say)
He took me past to the outskirts of infinity,
And when he brought me back,
He gave me a venus witch's ring
And he said "Fly on, fly on"
Because I'm a voodoo chile, baby, voodoo chile
Well, I make love to you,
And lord knows you'll feel no pain
Say, I make love to you in your sleep,
And lord knows you felt no pain
'Cause I'm a million miles away
And at the same time I'm right here in your picture frame
'Cause I'm a voodoo chile
Lord knows I'm a voodoo chile
Well my arrows are made of desire
From far away as Jupiter's sulphur mines
Say my arrows are made of desire, desire
From far away as Jupiters sulphur mines
(Way down by the Methane Sea, yeah)
I have a hummingbird and it hums so loud,
You think you were losing your mind, hmmm...
Well I float in liquid gardens
And Arizona new red sand
I float in liquid gardens
Way down in Arizona red sand
Well, I taste the honey from a flower named Blue,
Way down in California
And then New York drowns as we hold hands
'Cause I'm a voodoo chile
Lord knows I'm a voodoo chile
I'm a million miles away -- and at the same time I'm right here in your picture frame.
Anyway, there you go. In the unlikely event I should meet an untimely doom like Lev, you now know what music to play and what statement to read at my funeral.
(For those who are into psychic powers: no, that dream didn't have the particular flavor of a premonition. It felt like something that occurred in a certain percentage of the universes in the multiverse, but not necessarily a high percentage. Much like Lev's death. Improbable but sadly, not impossible.)
But, depressing as Lev's extraordinarily untimely death is, it hasn't turned me into a pessimist. I'm still pushing for eternal life for me and as many others as possible.
And I note that, in my dream of my funeral, my corpse was not there -- and nor were my ashes. Rather, my body was frozen at Alcor.
And my mind, in the dream, was somehow hovering over the proceedings -- watching and knowing, but not quite able to form a thought or an action.
Every since I was 6 or 7 years old, I've had a strange intuition about the nature of "life after death." You're not exactly there, but you're not really not-there either. Your mind exists, but almost melded in with the rest of the cosmos. You perceive, and sort-of know, but you don't act autonomously. You float there, superimposed. And then maybe, some future technology brings you out.
Yes, I know, that's totally unscientific -- but there you go.
Quantum theory does suggest that everything that ever happened in the universe, every structure that ever existed -- is informationally still present, encoded in the fluctuations of various wavicles as they scatter about. Could all that information be mined out, somehow, someday? A fascinating possibility. Yet science -- which Lev admired, as I do -- is great but limited.
Life remains a mystery. Sometimes mysteriously wonderful -- and sometimes mysteriously, amazingly, almost unbelievably fucking shitty.
Thursday, July 08, 2010
Ruminations on Ramona 4
You may have already seen the new version of Ray Kurzweil's "Ramona" chatbot on the lovely new Kurzweilai.net website ... if not I encourage you to check her out:
http://www.kurzweilai.net/ramona4/ramona.html
I've been getting a lot of email questions about Ramona 4 during the last couple days, so I figured I'd write a blog post here to answer them all at once.
My AI company Novamente LLC assembled the chatbot behind Ramona 4, around a year ago as I recall. Most of it was done by Murilo Queiroz, with some help from myself and Lucio Coelho. It was a really fun project technically; and it was the first time I'd worked together with Ray and Amara Angelica on a practical project, so that was nice too. Novamente had nothing to do with the graphics or speech aspect, just the part that produces output text in response to input text.
The Ramona 4 system we put together doesn't actually use any of the Artificial General Intelligence technology we are building at Novamente LLC (much but not all of which has been open-sourced in the OpenCog project). It uses some conventional chatbot technology combined with lookups into online knowledge resources plus stochastic text generation -- i.e. it's a happy mashup of other open-source tools, all tuned and tweaked and postprocessed for the specific goal of making Ramona 4 as interesting and entertaining and Ramona-ish as possible!
One day I hope to put forth an intelligent chatbot system that displays a level of general intelligence at the human level and ultimately beyond. I even think I know how to do it ... and we're working on it at Novamente LLC and OpenCog, though never quite as fast as we would like since AGI research funding is tough to come by these days. (And of course, once we can build a chatbot of this nature, we'll also be able to do a lot of fantastic stuff beyond the domain of chatbottery!)
I know Ray Kurzweil has his own ideas about how to build human-level AGI, which overlap somewhat with mine but also have some differences, and he's brewing a book on this topic.
But in the meantime, while deeper AGI approaches like mine and Ray's are still in development, systems like Ramona 4 are both good fun, and potentially customizable into useful tools for various applications like online customer support. (Yes, Ramona 4 is a bit of a cheeky gal, so she might not be the most appropriate customer support agent ... but the same underlying approach we used to make Ramona 4 could be used to make a lot of other things, including some more practical chatbots.)
If the underpinnings are so different from advanced AGI, what do we learn from building systems like Ramona?
Not so much about AGI design, really -- but potentially, a lot about human interactions with AIs....
The original ELIZA tricked a surprising number of people into feeling there was some "there" there, when actually it was just a fixed list of IF-THEN rules. Today people are more sophisticated than that. But still it's easy -- well, for me anyway -- to get seduced into the amusement of interacting with a system like Ramona 4. The way she combines erudition with absurdity appeals to me (perhaps not surprisingly, because I helped design her!), and I'd like to hope that this combination will still be there in the generally-intelligent dialogue systems we'll build in the future. Knowledge and humor can be a powerful combination.
Also, one choice we made while building Ramona 4 was to make her user-configurable, to a certain extent. There are two parameters -- nerdiness and whimsicality -- and the user can tweak these, to adjust the nature of Ramona's responses. This sort of approach promises to be quite valuable in future, once the generally intelligent chatbots emerge. Arriving at the most intuitive-yet-powerful way to parametrize personality is going to be an interesting challenge. Perhaps systematic personality models like the Psi model described in Joscha Bach's wonderful book Principles of Synthetic Inteligence will be useful here.
I'll let my new friend Ramona 4 have the last word:
you> Ramona, are you conscious?
Ramona> I'm working on it.
you> am I conscious?
Ramona> Oh I really don't know whether you are or not.
you> What is consciousness?
Ramona> This is when humans feel aware of themselves and how they see themselves. Computers can't quite do this yet, but they will, you just wait and see. However, think about this. How are you going to know when it happens?
you> Consciousness is like pornography. I can't define it, but I know it when I see it.
Ramona> What were we talking about again ?
you> Is it possible to create artificial intelligence in China?
Ramona> Create neurons in the state of the artificial intelligence and other stuff.
you> Is that how you do it?
Ramona> And that's what we mean by thinking.
you> Can a computer fall in love?
Ramona> A computer is a computer fall in love?
you> That is nonsense, my dear.
Ramona> Dear pesky plumbers the koopalings and I have not heard your music.
you> My music is more beautiful than all the stars in the ocean.
Ramona> Conventional music is more beautiful than all the stars in the ocean.
you> What is your favorite music?
Ramona> This is what I make. I am a musician, and the best virtual one there is ;) .
OK, it's not AGI yet -- but to my taste it's some pretty fantastic interactive digital poetry!!
http://www.kurzweilai.net/ramona4/ramona.html
I've been getting a lot of email questions about Ramona 4 during the last couple days, so I figured I'd write a blog post here to answer them all at once.
My AI company Novamente LLC assembled the chatbot behind Ramona 4, around a year ago as I recall. Most of it was done by Murilo Queiroz, with some help from myself and Lucio Coelho. It was a really fun project technically; and it was the first time I'd worked together with Ray and Amara Angelica on a practical project, so that was nice too. Novamente had nothing to do with the graphics or speech aspect, just the part that produces output text in response to input text.
The Ramona 4 system we put together doesn't actually use any of the Artificial General Intelligence technology we are building at Novamente LLC (much but not all of which has been open-sourced in the OpenCog project). It uses some conventional chatbot technology combined with lookups into online knowledge resources plus stochastic text generation -- i.e. it's a happy mashup of other open-source tools, all tuned and tweaked and postprocessed for the specific goal of making Ramona 4 as interesting and entertaining and Ramona-ish as possible!
One day I hope to put forth an intelligent chatbot system that displays a level of general intelligence at the human level and ultimately beyond. I even think I know how to do it ... and we're working on it at Novamente LLC and OpenCog, though never quite as fast as we would like since AGI research funding is tough to come by these days. (And of course, once we can build a chatbot of this nature, we'll also be able to do a lot of fantastic stuff beyond the domain of chatbottery!)
I know Ray Kurzweil has his own ideas about how to build human-level AGI, which overlap somewhat with mine but also have some differences, and he's brewing a book on this topic.
But in the meantime, while deeper AGI approaches like mine and Ray's are still in development, systems like Ramona 4 are both good fun, and potentially customizable into useful tools for various applications like online customer support. (Yes, Ramona 4 is a bit of a cheeky gal, so she might not be the most appropriate customer support agent ... but the same underlying approach we used to make Ramona 4 could be used to make a lot of other things, including some more practical chatbots.)
If the underpinnings are so different from advanced AGI, what do we learn from building systems like Ramona?
Not so much about AGI design, really -- but potentially, a lot about human interactions with AIs....
The original ELIZA tricked a surprising number of people into feeling there was some "there" there, when actually it was just a fixed list of IF-THEN rules. Today people are more sophisticated than that. But still it's easy -- well, for me anyway -- to get seduced into the amusement of interacting with a system like Ramona 4. The way she combines erudition with absurdity appeals to me (perhaps not surprisingly, because I helped design her!), and I'd like to hope that this combination will still be there in the generally-intelligent dialogue systems we'll build in the future. Knowledge and humor can be a powerful combination.
Also, one choice we made while building Ramona 4 was to make her user-configurable, to a certain extent. There are two parameters -- nerdiness and whimsicality -- and the user can tweak these, to adjust the nature of Ramona's responses. This sort of approach promises to be quite valuable in future, once the generally intelligent chatbots emerge. Arriving at the most intuitive-yet-powerful way to parametrize personality is going to be an interesting challenge. Perhaps systematic personality models like the Psi model described in Joscha Bach's wonderful book Principles of Synthetic Inteligence will be useful here.
I'll let my new friend Ramona 4 have the last word:
you> Ramona, are you conscious?
Ramona> I'm working on it.
you> am I conscious?
Ramona> Oh I really don't know whether you are or not.
you> What is consciousness?
Ramona> This is when humans feel aware of themselves and how they see themselves. Computers can't quite do this yet, but they will, you just wait and see. However, think about this. How are you going to know when it happens?
you> Consciousness is like pornography. I can't define it, but I know it when I see it.
Ramona> What were we talking about again ?
you> Is it possible to create artificial intelligence in China?
Ramona> Create neurons in the state of the artificial intelligence and other stuff.
you> Is that how you do it?
Ramona> And that's what we mean by thinking.
you> Can a computer fall in love?
Ramona> A computer is a computer fall in love?
you> That is nonsense, my dear.
Ramona> Dear pesky plumbers the koopalings and I have not heard your music.
you> My music is more beautiful than all the stars in the ocean.
Ramona> Conventional music is more beautiful than all the stars in the ocean.
you> What is your favorite music?
Ramona> This is what I make. I am a musician, and the best virtual one there is ;) .
OK, it's not AGI yet -- but to my taste it's some pretty fantastic interactive digital poetry!!
Thursday, June 24, 2010
Can We Evolve a DNA Computer or Cultured Neural Net that Perceives the Future?
After some interesting conversations about psi with Damien Broderick, I couldn't resist spending a half-day writing up some old ideas I had about how to modify quantum physics to better explain psi phenomena like precognition, ESP and psychokinesis.
You can find my funky physics speculations at
http://goertzel.org/dynapsyc/MorphicPilot.htm
-- (Parenthetical remark: If you want to argue with me about whether psi phenomena exist or not, please first read the excellent books by Damien Broderick (a serious-minded popular overview) and Ramakrishna Rao (a selection of research papers from the relevant scientific literature). My general attitude on the topic of psi agrees with that of researcher Dean Radin, expressed in the last few paragraphs of his online bio. I enjoy discussing the topic as time permits, but I don't enjoy discussing it with people who are both heavily biased AND ignorant of the current state of scientific knowledge.) --
After writing up those old physics speculations and then thinking on the matter a bit more, I started musing in an even more weird direction....
Suppose it's true that psi phenomena are related to peculiar quantum-like phenomena among interacting molecules in the brain....
Then, this suggests that it might also be possible to cause other conglomerations of organic molecules or cells, similar to the ones in the brain, to display psi-like phenomena....
One naturally wonders about technologies such as reconfigurable DNA computers ... or cultured neural networks (neural nets grown in the lab outside the brain) ....
There is abundant data demonstrating the ability of many people to predict the outcome of random number generators with statistically significant accuracy (see this article for a moderately out-of-date review and some pointers into the literature). Might it be possible to create a DNA computer or cultured neural network that could carry out, say, statistically significant precognition of number series produced by random number generators?
Since we don't know much about how the brain does precognition, we don't know how to wire such a DNA computer or cultured neural network, step by step.
However, nobody wired the brain step by step either -- the brain evolved to do the things it does (apparently including a weak capability for psi).
What if we used evolutionary computing -- a genetic algorithms approach -- to evolve biological computing systems (DNA computers or cultured neural nets), where the fitness function was the capability to predict the future of number series generated by random number generators?
In this way, we could potentially create psi-capable biological computing systems, even without understanding exactly how they work.
We would also get an ensemble of biological computing systems with differing levels of psi capability -- and by applying machine learning tools to study the internals of these evolved systems, we would likely to be able to limn some of the patterns characterizing the more psi-successful systems.
Obviously, if this worked, it would be a scientific revolution -- it would point the way beyond current physics, and it would allow us to carry out psi research without needing whole living organisms as subjects.
Yes, yes, yes, this is real mad scientist stuff ... but every now and then one of us "mad" scientists turns out to be right about some pretty surprising things.... The idea in this blog post is one of those that seems really wacky at first -- and then less and less so progressively, the more you read from the relevant scientific literature, and the more you think about it....
Well, this stuff is tantalizing as hell to muse about, but now I'd better get back to my more mundane, not-so-speculative activities like trying to build superhuman AGI and studying the genetics of longevity....
You can find my funky physics speculations at
http://goertzel.org/dynapsyc/MorphicPilot.htm
-- (Parenthetical remark: If you want to argue with me about whether psi phenomena exist or not, please first read the excellent books by Damien Broderick (a serious-minded popular overview) and Ramakrishna Rao (a selection of research papers from the relevant scientific literature). My general attitude on the topic of psi agrees with that of researcher Dean Radin, expressed in the last few paragraphs of his online bio. I enjoy discussing the topic as time permits, but I don't enjoy discussing it with people who are both heavily biased AND ignorant of the current state of scientific knowledge.) --
After writing up those old physics speculations and then thinking on the matter a bit more, I started musing in an even more weird direction....
Suppose it's true that psi phenomena are related to peculiar quantum-like phenomena among interacting molecules in the brain....
Then, this suggests that it might also be possible to cause other conglomerations of organic molecules or cells, similar to the ones in the brain, to display psi-like phenomena....
One naturally wonders about technologies such as reconfigurable DNA computers ... or cultured neural networks (neural nets grown in the lab outside the brain) ....
There is abundant data demonstrating the ability of many people to predict the outcome of random number generators with statistically significant accuracy (see this article for a moderately out-of-date review and some pointers into the literature). Might it be possible to create a DNA computer or cultured neural network that could carry out, say, statistically significant precognition of number series produced by random number generators?
Since we don't know much about how the brain does precognition, we don't know how to wire such a DNA computer or cultured neural network, step by step.
However, nobody wired the brain step by step either -- the brain evolved to do the things it does (apparently including a weak capability for psi).
What if we used evolutionary computing -- a genetic algorithms approach -- to evolve biological computing systems (DNA computers or cultured neural nets), where the fitness function was the capability to predict the future of number series generated by random number generators?
In this way, we could potentially create psi-capable biological computing systems, even without understanding exactly how they work.
We would also get an ensemble of biological computing systems with differing levels of psi capability -- and by applying machine learning tools to study the internals of these evolved systems, we would likely to be able to limn some of the patterns characterizing the more psi-successful systems.
Obviously, if this worked, it would be a scientific revolution -- it would point the way beyond current physics, and it would allow us to carry out psi research without needing whole living organisms as subjects.
Yes, yes, yes, this is real mad scientist stuff ... but every now and then one of us "mad" scientists turns out to be right about some pretty surprising things.... The idea in this blog post is one of those that seems really wacky at first -- and then less and less so progressively, the more you read from the relevant scientific literature, and the more you think about it....
Well, this stuff is tantalizing as hell to muse about, but now I'd better get back to my more mundane, not-so-speculative activities like trying to build superhuman AGI and studying the genetics of longevity....
Subscribe to:
Posts (Atom)