A recent article by Bill Hibbard, myself and other colleagues on the value of openness in
AI development (linked to a petition in favor of transparent AI development and deployment) has, predictably, gotten a bunch of comments from
people alarmed at the potential riskiness of putting AGI designs and
code in the hands of the general public.
Another complaint
from many commenters has been that there is so much profit in AGI
that big companies will inevitably dominate it, buying any small
actors who make significant innovations.
The second complaint
has a more obvious counter-argument – simply that not everyone can
be bought by big companies, either because
- they are already rich due to non-AI stuff, like the folks behind OpenAI
- they are “crazy / idealistic”, like those of us behind OpenCog
So if some folks who
cannot be bought make enough progress to seed a transparent and open
source AGI movement that takes off fast enough, it may beat the big
tech companies. Linux provides an inexact but meaningful analogue;
in some important ways it's beating the big companies at OS
development. And many of the individuals behind Linux are too
crazy/idealistic to be bought by big companies.
The counterargument
to the first complaint is slightly subtler, but has some important
ingredients in common with the “seed a movement” counterargument
to the second complaint. Saying “it's dangerous to give AGI to
bad guys” overlooks the complexity of the likely processes
underlying the development of AGI, and the relation of these
processes to the nature of the AGI itself and the social networks in which emerging AGIs are embedded.
In this blog post I
will explore some of this complexity. This is somewhat of a
preliminary document and these ideas will likely get presented in a
better organized and more systematic fashion later, somewhere or other....
First: it is true
that, given any specific AGI system, it MIGHT be safer to keep that
particular AGI system in the hands of a specific chosen group of
beneficial actors, than to spread knowledge of that AGI system
widely. Whether this is actually true in a particular instance
depends on various factors including:
- how beneficial that chosen group actually is
- the number and nature of “smart but malevolent or irresponsible” other parties on the scene
- how much resources the AGI requires to run effectively
- whether the rest of the world will get annoyed and start a fight about having the secrets of AGI kept from it (and then who is likely to win that fight)
- etc.
However, looking at
it this way overlooks many things, including the dependency between
two factors:
- the degree of openness with which an AGI system is developed
- the various properties (e.g. robustness and general beneficial-ness) of that AGI system
One could argue that
the RIGHT small, elite, closed group is more likely to develop a
robust and beneficial AGI system than a large, distributed, diverse
motley crew of participants (such as tends to emerge in a successful
OSS project community). On the other hand, this argument seems very
weak, because in an open source setting, there is still the
opportunity for altruistic smart, elite group to fork existing
codebases and create their own separate version, leveraging the work
done by others but perfecting it according to their own ideas and
aesthetics. One could argue that closed-source provides more
incentives for smart, elite groups to participate in projects. But
again this is weak, given the evidence of well-funded OSS AI
initiatives like OpenAI, and given the high level of technical
strength of many individuals in the broader OSS community. There are
more big proprietary projects than big OSS projects out there. But
the big OSS projects are generally very high in quality.
Commercial projects
have historically done better at user interface refinement than OSS
projects. But creating AGI is arguably more like building an OS
kernel or a machine learning library than like building a user
interface – i.e. it's complex and technically subtle, requiring
deep expertise to make real progress. This is the kind of thing the
OSS world has been good at. (Indeed, similar to user interfaces, we
need AGIs to respond to the various peculiarities of human nature.
But we need the AGI to learn this kind of response, we don't want to
code the particulars of human nature into the AGI one by one. And
this kind of learning appears to require algorithms that appear
highly technical and tricky given the current state of science.)
My own feeling is
that an open and transparent modality is much more likely to lead to
a robust and beneficial AGI. This is because there will be such a
diverse group of smart people working on it. And because the group
of people working on it will not be biased by having specific
commercial goals. Even when the business goals underlying a
commercial AGI system are not especially nefarious nor contradictory
to the goal of making broadly beneficial AGI – nevertheless, the
existence of specific commercial goals will bias the thinking of the
people involved in a certain direction, leading them to overlook
certain promising directions and also certain risks.
As Bill Hibbard
points out in his recent follow-up article, the “is open versus
closed AGI better” debate ties in closely with differing ideas
about AGI design and the likely process via which AGI will develop.
If one believes there is going to be a relatively small/simple AGI
design, which will give anyone who “knows the trick” dramatically
superior performance to anything that came before it – then there
is a reasonable argument for keeping this trick secret, assuming one
trusts the group that holds the secret. If one believes that the
first powerful AGI is likely to be more complex and heterogeneous,
emergent from the combination of a large number of different software
components carrying out different functions in different ways, then
there is less argument for keeping such systems secret as they
develop.
For one thing, in
the latter “big emergent combination” scenario, secrets about AGI
design will not likely be well kept anyway. Big tech companies are
far outpacing top-secret underground government labs in AI
development, and this trend seems likely to continue; but big
companies have ongoing employee turnover and tend not to be extremely
good at keeping secrets for long periods of time. If AGI is going to
emerge via a pathway that requires years of effort and incremental
improvements, then the in-process AGI system is bound to leak out
even if it's developed in a big-company lab. (Whether a top-secret
government lab would be any better at keeping a complex AGI design
secret is a different question. I doubt it; there are plenty of
spies and double agents about....)
For another thing, a
complex, heterogeneous system is exactly the sort of thing that a
large, diverse community has a lot to contribute to. Parts of such
a system that are not especially critical to any company's business
model, can nonetheless get loving care from some brilliant, focused
academic or hacker community.
In principle, of
course, if a company or government were rich enough and ambitious
enough, they could buy an almost arbitrarily diverse development community.
Taking this to the ultimate extent one has a fascist-type model
where some company or government agency rigidly controls everyone in
the world -- a Google-archic government, or whatever. But in practice any company or government agency seems
to only be able to acquire relatively limited resources, not
sufficient to enable them to fund top-notch teams working on
peripheral aspects of their AI projects.
So given all the
above, it seems we may well have a choice between
- a worse (less robust, less richly and generally intelligent) AGI that is created and owned by some closed group
- a better (more robust, more richly and generally intelligent) AGI that is created and deployed in an open way, and not owned by anyone
Given this kind of
choice, the risk of a nasty group of actors doing something bad with
an AGI would not be the decisive point. Rather, we need to look at
multiple options such as
- the odds of a nasty group getting ahold of an AGI and things going awry
- the odds of a group with broadly beneficial goals in mind getting ahold of an AGI and things going awry
- the odds of a group with non-malevolent but relatively narrow (e.g. commercial or national-security) goals getting ahold of an AGI and things going awry
On the face of it,
Problem 1 would seem more likely to occur with the open approach.
But if open-ness tends to lead to more robust and beneficial AGIs (as
I strongly suspect is the case) then Problems 2 and especially 3 are
more likely to occur in the closed case.
One must bear in
mind also that there are many ways for things to go awry. For
instance, things can go awry due to what an AGI directly does when
deployed in the world. Or, things can go awry due to the world's
reaction to what an AGI does when deployed in the world. This is a
point Bill Hibbard has highlighted in his response. A closed
commercial AGI will more likely be viewed as manipulating people for
a the good of a small elite group, and hence more likely to arouse
public ire and cause potential issues (such as hackers messing with
the AGI, for that matter). A closed military AGI has strong
potential to lead to arms-race scenarios.
Summing up, then -- I don't claim to
have a simple, knockout argument why transparent, open AGI is better.
But I want to emphasize that the apparent simple, knockout argument
why closed AGI is better, is simply delusory. Saying “closed AI is
better because with open AI, bad guys can take it and kill us all”
is simply sweeping copious complexities of actual reality under the
rug. It's roughly analogous to saying “having more weapons is
better because then it's possible to blow up a larger class of
enemies.” The latter problematically overlooks arms-race
phenomena, in which the number of weapons possessed by one group,
affects the number of weapons possessed by another group; and also
psychological phenomena via which both adversarial and beneficial
attitudes tend to spread. The former problematically overlooks the
dependencies between the open vs. closed choice and the nature of the
AGI to be developed, and the nature of the social network in which
the AGI gets released.
I have considered a bunch of complexly interdependent factors, in the above. They're all quite clear in my own head, but I worry that due to typing these notes hastily, I may not have made it all that clear to the reader. Some futurists have
proposed a Bayesian decision-tree approach to describing complex
future situations involving subtle choices. It might be interesting
to make such a tree regarding closed vs. transparent AGI, based on
the factors highlighted in this blog post along with any others that
come up. This might be a more effective way of clearly outlining all the relevant issues and their cross-relationships.
