Transparent AGI: Benefits and Risks

A recent article by Bill Hibbard, myself and other colleagues on the value of openness in AI development (linked to a petition in favor of transparent AI development and deployment) has, predictably, gotten a bunch of comments from people alarmed at the potential riskiness of putting AGI designs and code in the hands of the general public.

Another complaint from many commenters has been that there is so much profit in AGI that big companies will inevitably dominate it, buying any small actors who make significant innovations.

The second complaint has a more obvious counter-argument – simply that not everyone can be bought by big companies, either because

• they are already rich due to non-AI stuff, like the folks behind OpenAI
• they are “crazy / idealistic”, like those of us behind OpenCog

So if some folks who cannot be bought make enough progress to seed a transparent and open source AGI movement that takes off fast enough, it may beat the big tech companies. Linux provides an inexact but meaningful analogue; in some important ways it's beating the big companies at OS development. And many of the individuals behind Linux are too crazy/idealistic to be bought by big companies.

The counterargument to the first complaint is slightly subtler, but has some important ingredients in common with the “seed a movement” counterargument to the second complaint. Saying “it's dangerous to give AGI to bad guys” overlooks the complexity of the likely processes underlying the development of AGI, and the relation of these processes to the nature of the AGI itself and the social networks in which emerging AGIs are embedded.

In this blog post I will explore some of this complexity. This is somewhat of a preliminary document and these ideas will likely get presented in a better organized and more systematic fashion later, somewhere or other....

First: it is true that, given any specific AGI system, it MIGHT be safer to keep that particular AGI system in the hands of a specific chosen group of beneficial actors, than to spread knowledge of that AGI system widely. Whether this is actually true in a particular instance depends on various factors including:

• how beneficial that chosen group actually is
• the number and nature of “smart but malevolent or irresponsible” other parties on the scene
• how much resources the AGI requires to run effectively
• whether the rest of the world will get annoyed and start a fight about having the secrets of AGI kept from it (and then who is likely to win that fight)
• etc.

However, looking at it this way overlooks many things, including the dependency between two factors:

• the degree of openness with which an AGI system is developed
• the various properties (e.g. robustness and general beneficial-ness) of that AGI system

One could argue that the RIGHT small, elite, closed group is more likely to develop a robust and beneficial AGI system than a large, distributed, diverse motley crew of participants (such as tends to emerge in a successful OSS project community). On the other hand, this argument seems very weak, because in an open source setting, there is still the opportunity for altruistic smart, elite group to fork existing codebases and create their own separate version, leveraging the work done by others but perfecting it according to their own ideas and aesthetics. One could argue that closed-source provides more incentives for smart, elite groups to participate in projects. But again this is weak, given the evidence of well-funded OSS AI initiatives like OpenAI, and given the high level of technical strength of many individuals in the broader OSS community. There are more big proprietary projects than big OSS projects out there. But the big OSS projects are generally very high in quality.

Commercial projects have historically done better at user interface refinement than OSS projects. But creating AGI is arguably more like building an OS kernel or a machine learning library than like building a user interface – i.e. it's complex and technically subtle, requiring deep expertise to make real progress. This is the kind of thing the OSS world has been good at. (Indeed, similar to user interfaces, we need AGIs to respond to the various peculiarities of human nature. But we need the AGI to learn this kind of response, we don't want to code the particulars of human nature into the AGI one by one. And this kind of learning appears to require algorithms that appear highly technical and tricky given the current state of science.)

My own feeling is that an open and transparent modality is much more likely to lead to a robust and beneficial AGI. This is because there will be such a diverse group of smart people working on it. And because the group of people working on it will not be biased by having specific commercial goals. Even when the business goals underlying a commercial AGI system are not especially nefarious nor contradictory to the goal of making broadly beneficial AGI – nevertheless, the existence of specific commercial goals will bias the thinking of the people involved in a certain direction, leading them to overlook certain promising directions and also certain risks.

As Bill Hibbard points out in his recent follow-up article, the “is open versus closed AGI better” debate ties in closely with differing ideas about AGI design and the likely process via which AGI will develop. If one believes there is going to be a relatively small/simple AGI design, which will give anyone who “knows the trick” dramatically superior performance to anything that came before it – then there is a reasonable argument for keeping this trick secret, assuming one trusts the group that holds the secret. If one believes that the first powerful AGI is likely to be more complex and heterogeneous, emergent from the combination of a large number of different software components carrying out different functions in different ways, then there is less argument for keeping such systems secret as they develop.

For one thing, in the latter “big emergent combination” scenario, secrets about AGI design will not likely be well kept anyway. Big tech companies are far outpacing top-secret underground government labs in AI development, and this trend seems likely to continue; but big companies have ongoing employee turnover and tend not to be extremely good at keeping secrets for long periods of time. If AGI is going to emerge via a pathway that requires years of effort and incremental improvements, then the in-process AGI system is bound to leak out even if it's developed in a big-company lab. (Whether a top-secret government lab would be any better at keeping a complex AGI design secret is a different question. I doubt it; there are plenty of spies and double agents about....)

For another thing, a complex, heterogeneous system is exactly the sort of thing that a large, diverse community has a lot to contribute to. Parts of such a system that are not especially critical to any company's business model, can nonetheless get loving care from some brilliant, focused academic or hacker community.

In principle, of course, if a company or government were rich enough and ambitious enough, they could buy an almost arbitrarily diverse development community. Taking this to the ultimate extent one has a fascist-type model where some company or government agency rigidly controls everyone in the world -- a Google-archic government, or whatever. But in practice any company or government agency seems to only be able to acquire relatively limited resources, not sufficient to enable them to fund top-notch teams working on peripheral aspects of their AI projects.

So given all the above, it seems we may well have a choice between

• a worse (less robust, less richly and generally intelligent) AGI that is created and owned by some closed group
• a better (more robust, more richly and generally intelligent) AGI that is created and deployed in an open way, and not owned by anyone

Given this kind of choice, the risk of a nasty group of actors doing something bad with an AGI would not be the decisive point. Rather, we need to look at multiple options such as

1. the odds of a nasty group getting ahold of an AGI and things going awry
2. the odds of a group with broadly beneficial goals in mind getting ahold of an AGI and things going awry
3. the odds of a group with non-malevolent but relatively narrow (e.g. commercial or national-security) goals getting ahold of an AGI and things going awry

On the face of it, Problem 1 would seem more likely to occur with the open approach. But if open-ness tends to lead to more robust and beneficial AGIs (as I strongly suspect is the case) then Problems 2 and especially 3 are more likely to occur in the closed case.

One must bear in mind also that there are many ways for things to go awry. For instance, things can go awry due to what an AGI directly does when deployed in the world. Or, things can go awry due to the world's reaction to what an AGI does when deployed in the world. This is a point Bill Hibbard has highlighted in his response. A closed commercial AGI will more likely be viewed as manipulating people for a the good of a small elite group, and hence more likely to arouse public ire and cause potential issues (such as hackers messing with the AGI, for that matter). A closed military AGI has strong potential to lead to arms-race scenarios.

Summing up, then -- I don't claim to have a simple, knockout argument why transparent, open AGI is better. But I want to emphasize that the apparent simple, knockout argument why closed AGI is better, is simply delusory. Saying “closed AI is better because with open AI, bad guys can take it and kill us all” is simply sweeping copious complexities of actual reality under the rug. It's roughly analogous to saying “having more weapons is better because then it's possible to blow up a larger class of enemies.” The latter problematically overlooks arms-race phenomena, in which the number of weapons possessed by one group, affects the number of weapons possessed by another group; and also psychological phenomena via which both adversarial and beneficial attitudes tend to spread. The former problematically overlooks the dependencies between the open vs. closed choice and the nature of the AGI to be developed, and the nature of the social network in which the AGI gets released.

I have considered a bunch of complexly interdependent factors, in the above.  They're all quite clear in my own head, but I worry that due to typing these notes hastily, I may not have made it all that clear to the reader.   Some futurists have proposed a Bayesian decision-tree approach to describing complex future situations involving subtle choices. It might be interesting to make such a tree regarding closed vs. transparent AGI, based on the factors highlighted in this blog post along with any others that come up.   This might be a more effective way of clearly outlining all the relevant issues and their cross-relationships.