Go ahead, read the whole thing.<\/p>\n
\n“Within thirty years, we will have the technological means to create superhuman intelligence. Shortly after, the human era will be ended.<\/em><\/p>\n
Is such progress avoidable? If not to be avoided, can events be guided so that we may survive? These questions are investigated. Some possible answers (and some further dangers) are presented.”<\/em><\/p>\n<\/blockquote>\n
<\/p>\n
Vernor Vinge: The Singularity<\/h1>\n
Vernor Vinge Department of Mathematical Sciences San Diego State
\nUniversity
(c) 1993 by Vernor Vinge <\/p>\nThis article may be reproduced for noncommercial purposes if it is copied
\nin its entirety, including this notice.<\/i><\/p>\n\nThe original version of this article was presented at the VISION-21 Symposium
\nsponsored by NASA Lewis Research Center and the Ohio Aerospace Institute, March
\n30-31, 1993.<\/p>\nA slightly changed version appeared in the Winter 1993 issue of _Whole Earth
\nReview_. <\/p>\n<\/blockquote>\nAbstract<\/h3>\n
Within thirty years, we will have the technological means to create
\nsuperhuman intelligence. Shortly after, the human era will be ended.<\/p>\nIs such progress avoidable? If not to be avoided, can events be guided so
\nthat we may survive? These questions are investigated. Some possible answers
\n(and some further dangers) are presented.<\/p>\nWhat is The Singularity?<\/h3>\n
The acceleration of technological progress has been the central feature of
\nthis century. I argue in this paper that we are on the edge of change comparable
\nto the rise of human life on Earth. The precise cause of this change is the
\nimminent creation by technology of entities with greater than human
\nintelligence. There are several means by which science may achieve this
\nbreakthrough (and this is another reason for having confidence that the event
\nwill occur): <\/p>\n\n
- There may be developed computers that are “awake” and superhumanly
\nintelligent. (To date, there has been much controversy as to whether we can
\ncreate human equivalence in a machine. But if the answer is “yes, we can”, then
\nthere is little doubt that beings more intelligent can be constructed shortly
\nthereafter.)<\/p>\n- Large computer networks (and their associated users) may “wake up” as a
\nsuperhumanly intelligent entity.<\/p>\n- Computer\/human interfaces may become so intimate that users may reasonably
\nbe considered superhumanly intelligent.<\/p>\n- Biological science may provide means to improve natural human intellect.\n<\/li>\n<\/ul>\n
The first three possibilities depend in large part on improvements in
\ncomputer hardware. Progress in computer hardware has followed an amazingly
\nsteady curve in the last few decades [17]. Based largely on this trend, I
\nbelieve that the creation of greater than human intelligence will occur during
\nthe next thirty years. (Charles Platt [20] has pointed out that AI enthusiasts
\nhave been making claims like this for the last thirty years. Just so I’m not
\nguilty of a relative-time ambiguity, let me more specific: I’ll be surprised if
\nthis event occurs before 2005 or after 2030.)<\/p>\nWhat are the consequences of this event? When greater-than-human intelligence
\ndrives progress, that progress will be much more rapid. In fact, there seems no
\nreason why progress itself would not involve the creation of still more
\nintelligent entities — on a still-shorter time scale. The best analogy that I
\nsee is with the evolutionary past: Animals can adapt to problems and make
\ninventions, but often no faster than natural selection can do its work — the
\nworld acts as its own simulator in the case of natural selection. We humans have
\nthe ability to internalize the world and conduct “what if’s” in our heads; we
\ncan solve many problems thousands of times faster than natural selection. Now,
\nby creating the means to execute those simulations at much higher speeds, we are
\nentering a regime as radically different from our human past as we humans are
\nfrom the lower animals.<\/p>\nFrom the human point of view this change will be a throwing away of all the
\nprevious rules, perhaps in the blink of an eye, an exponential runaway beyond
\nany hope of control. Developments that before were thought might only happen in
\n“a million years” (if ever) will likely happen in the next century. (In [5],
\nGreg Bear paints a picture of the major changes happening in a matter of
\nhours.)<\/p>\nI think it’s fair to call this event a singularity (“the Singularity” for the
\npurposes of this paper). It is a point where our old models must be discarded
\nand a new reality rules. As we move closer to this point, it will loom vaster
\nand vaster over human affairs till the notion becomes a commonplace. Yet when it
\nfinally happens it may still be a great surprise and a greater unknown. In the
\n1950s there were very few who saw it: Stan Ulam [28] paraphrased John von
\nNeumann as saying:<\/p>\n\nOne conversation centered on the ever accelerating progress of technology and
\nchanges in the mode of human life, which gives the appearance of approaching
\nsome essential singularity in the history of the race beyond which human
\naffairs, as we know them, could not continue.<\/p>\n<\/blockquote>\nVon Neumann even uses the term singularity, though it appears he is thinking
\nof normal progress, not the creation of superhuman intellect. (For me, the
\nsuperhumanity is the essence of the Singularity. Without that we would get a
\nglut of technical riches, never properly absorbed (see [25]).)<\/p>\nIn the 1960s there was recognition of some of the implications of superhuman
\nintelligence. I. J. Good wrote [11]:<\/p>\n\nLet an ultraintelligent machine be defined as a machine that can far surpass
\nall the intellectual activities of any any man however clever. Since the design
\nof machines is one of these intellectual activities, an ultraintelligent machine
\ncould design even better machines; there would then unquestionably be an
\n“intelligence explosion,” and the intelligence of man would be left far behind.
\nThus the first ultraintelligent machine is the _last_ invention that man need
\never make, provided that the machine is docile enough to tell us how to keep it
\nunder control.
…
It is more probable than not that, within the
\ntwentieth century, an ultraintelligent machine will be built and that it will be
\nthe last invention that man need make.<\/p>\n<\/blockquote>\nGood has captured the essence of the runaway, but does not pursue its most
\ndisturbing consequences. Any intelligent machine of the sort he describes would
\nnot be humankind’s “tool” — any more than humans are the tools of rabbits or
\nrobins or chimpanzees.<\/p>\nThrough the ’60s and ’70s and ’80s, recognition of the cataclysm spread [29]
\n[1] [31] [5]. Perhaps it was the science-fiction writers who felt the first
\nconcrete impact. After all, the “hard” science-fiction writers are the ones who
\ntry to write specific stories about all that technology may do for us. More and
\nmore, these writers felt an opaque wall across the future. Once, they could put
\nsuch fantasies millions of years in the future [24]. Now they saw that their
\nmost diligent extrapolations resulted in the unknowable … soon. Once, galactic
\nempires might have seemed a Post-Human domain. Now, sadly, even interplanetary
\nones are.<\/p>\nWhat about the ’90s and the ’00s and the ’10s, as we slide toward the edge?
\nHow will the approach of the Singularity spread across the human world view? For
\na while yet, the general critics of machine sapience will have good press. After
\nall, till we have hardware as powerful as a human brain it is probably foolish
\nto think we’ll be able to create human equivalent (or greater) intelligence.
\n(There is the far-fetched possibility that we could make a human equivalent out
\nof less powerful hardware, if we were willing to give up speed, if we were
\nwilling to settle for an artificial being who was literally slow [30]. But it’s
\nmuch more likely that devising the software will be a tricky process, involving
\nlots of false starts and experimentation. If so, then the arrival of self-aware
\nmachines will not happen till after the development of hardware that is
\nsubstantially more powerful than humans’ natural equipment.)<\/p>\nBut as time passes, we should see more symptoms. The dilemma felt by science
\nfiction writers will be perceived in other creative endeavors. (I have heard
\nthoughtful comic book writers worry about how to have spectacular effects when
\neverything visible can be produced by the technologically commonplace.) We will
\nsee automation replacing higher and higher level jobs. We have tools right now
\n(symbolic math programs, cad\/cam) that release us from most low-level drudgery.
\nOr put another way: The work that is truly productive is the domain of a
\nsteadily smaller and more elite fraction of humanity. In the coming of the
\nSingularity, we are seeing the predictions of _true_ technological unemployment
\nfinally come true.<\/p>\nAnother symptom of progress toward the Singularity: ideas themselves should
\nspread ever faster, and even the most radical will quickly become commonplace.
\nWhen I began writing science fiction in the middle ’60s, it seemed very easy to
\nfind ideas that took decades to percolate into the cultural consciousness; now
\nthe lead time seems more like eighteen months. (Of course, this could just be me
\nlosing my imagination as I get old, but I see the effect in others too.) Like
\nthe shock in a compressible flow, the Singularity moves closer as we accelerate
\nthrough the critical speed.<\/p>\nAnd what of the arrival of the Singularity itself? What can be said of its
\nactual appearance? Since it involves an intellectual runaway, it will probably
\noccur faster than any technical revolution seen so far. The precipitating event
\nwill likely be unexpected — perhaps even to the researchers involved. (“But all
\nour previous models were catatonic! We were just tweaking some parameters….”)
\nIf networking is widespread enough (into ubiquitous embedded systems), it may
\nseem as if our artifacts as a whole had suddenly wakened.<\/p>\nAnd what happens a month or two (or a day or two) after that? I have only
\nanalogies to point to: The rise of humankind. We will be in the Post-Human era.
\nAnd for all my rampant technological optimism, sometimes I think I’d be more
\ncomfortable if I were regarding these transcendental events from one thousand
\nyears remove … instead of twenty.<\/p>\nCan the Singularity be Avoided?<\/h3>\n
Well, maybe it won’t happen at all: Sometimes I try to imagine the symptoms
\nthat we should expect to see if the Singularity is not to develop. There are the
\nwidely respected arguments of Penrose [19] and Searle [22] against the
\npracticality of machine sapience. In August of 1992, Thinking Machines
\nCorporation held a workshop to investigate the question “How We Will Build a
\nMachine that Thinks” [27]. As you might guess from the workshop’s title, the
\nparticipants were not especially supportive of the arguments against machine
\nintelligence. In fact, there was general agreement that minds can exist on
\nnonbiological substrates and that algorithms are of central importance to the
\nexistence of minds. However, there was much debate about the raw hardware power
\nthat is present in organic brains. A minority felt that the largest 1992
\ncomputers were within three orders of magnitude of the power of the human brain.
\nThe majority of the participants agreed with Moravec’s estimate [17] that we are
\nten to forty years away from hardware parity. And yet there was another minority
\nwho pointed to [7] [21], and conjectured that the computational competence of
\nsingle neurons may be far higher than generally believed. If so, our present
\ncomputer hardware might be as much as _ten_ orders of magnitude short of the
\nequipment we carry around in our heads. If this is true (or for that matter, if
\nthe Penrose or Searle critique is valid), we might never see a Singularity.
\nInstead, in the early ’00s we would find our hardware performance curves
\nbeginning to level off — this because of our inability to automate the design
\nwork needed to support further hardware improvements. We’d end up with some
\n_very_ powerful hardware, but without the ability to push it further. Commercial
\ndigital signal processing might be awesome, giving an analog appearance even to
\ndigital operations, but nothing would ever “wake up” and there would never be
\nthe intellectual runaway which is the essence of the Singularity. It would
\nlikely be seen as a golden age … and it would also be an end of progress. This
\nis very like the future predicted by Gunther Stent. In fact, on page 137 of
\n[25], Stent explicitly cites the development of transhuman intelligence as a
\nsufficient condition to break his projections.<\/p>\nBut if the technological Singularity can happen, it will. Even if all the
\ngovernments of the world were to understand the “threat” and be in deadly fear
\nof it, progress toward the goal would continue. In fiction, there have been
\nstories of laws passed forbidding the construction of “a machine in the likeness
\nof the human mind” [13]. In fact, the competitive advantage — economic,
\nmilitary, even artistic — of every advance in automation is so compelling that
\npassing laws, or having customs, that forbid such things merely assures that
\nsomeone else will get them first.<\/p>\nEric Drexler [8] has provided spectacular insights about how far technical
\nimprovement may go. He agrees that superhuman intelligences will be available in
\nthe near future — and that such entities pose a threat to the human status quo.
\nBut Drexler argues that we can confine such transhuman devices so that their
\nresults can be examined and used safely. This is I. J. Good’s ultraintelligent
\nmachine, with a dose of caution. I argue that confinement is intrinsically
\nimpractical. For the case of physical confinement: Imagine yourself locked in
\nyour home with only limited data access to the outside, to your masters. If
\nthose masters thought at a rate — say — one million times slower than you,
\nthere is little doubt that over a period of years (your time) you could come up
\nwith “helpful advice” that would incidentally set you free. (I call this “fast
\nthinking” form of superintelligence “weak superhumanity”. Such a “weakly
\nsuperhuman” entity would probably burn out in a few weeks of outside time.
\n“Strong superhumanity” would be more than cranking up the clock speed on a
\nhuman-equivalent mind. It’s hard to say precisely what “strong superhumanity”
\nwould be like, but the difference appears to be profound. Imagine running a dog
\nmind at very high speed. Would a thousand years of doggy living add up to any
\nhuman insight? (Now if the dog mind were cleverly rewired and _then_ run at high
\nspeed, we might see something different….) Many speculations about
\nsuperintelligence seem to be based on the weakly superhuman model. I believe
\nthat our best guesses about the post-Singularity world can be obtained by
\nthinking on the nature of strong superhumanity. I will return to this point
\nlater in the paper.)<\/p>\nAnother approach to confinement is to build _rules_ into the mind of the
\ncreated superhuman entity (for example, Asimov’s Laws [3]). I think that any
\nrules strict enough to be effective would also produce a device whose ability
\nwas clearly inferior to the unfettered versions (and so human competition would
\nfavor the development of the those more dangerous models). Still, the Asimov
\ndream is a wonderful one: Imagine a willing slave, who has 1000 times your
\ncapabilities in every way. Imagine a creature who could satisfy your every safe
\nwish (whatever that means) and still have 99.9% of its time free for other
\nactivities. There would be a new universe we never really understood, but filled
\nwith benevolent gods (though one of _my_ wishes might be to become one of
\nthem).<\/p>\nIf the Singularity can not be prevented or confined, just how bad could the
\nPost-Human era be? Well … pretty bad. The physical extinction of the human
\nrace is one possibility. (Or as Eric Drexler put it of nanotechnology: Given all
\nthat such technology can do, perhaps governments would simply decide that they
\nno longer need citizens!). Yet physical extinction may not be the scariest
\npossibility. Again, analogies: Think of the different ways we relate to animals.
\nSome of the crude physical abuses are implausible, yet…. In a Post-Human world
\nthere would still be plenty of niches where human equivalent automation would be
\ndesirable: embedded systems in autonomous devices, self-aware daemons in the
\nlower functioning of larger sentients. (A strongly superhuman intelligence would
\nlikely be a Society of Mind [16] with some very competent components.) Some of
\nthese human equivalents might be used for nothing more than digital signal
\nprocessing. They would be more like whales than humans. Others might be very
\nhuman-like, yet with a one-sidedness, a _dedication_ that would put them in a
\nmental hospital in our era. Though none of these creatures might be
\nflesh-and-blood humans, they might be the closest things in the new enviroment
\nto what we call human now. (I. J. Good had something to say about this, though
\nat this late date the advice may be moot: Good [12] proposed a “Meta-Golden
\nRule”, which might be paraphrased as “Treat your inferiors as you would be
\ntreated by your superiors.” It’s a wonderful, paradoxical idea (and most of my
\nfriends don’t believe it) since the game-theoretic payoff is so hard to
\narticulate. Yet if we were able to follow it, in some sense that might say
\nsomething about the plausibility of such kindness in this universe.)<\/p>\nI have argued above that we cannot prevent the Singularity, that its coming
\nis an inevitable consequence of the humans’ natural competitiveness and the
\npossibilities inherent in technology. And yet … we are the initiators. Even
\nthe largest avalanche is triggered by small things. We have the freedom to
\nestablish initial conditions, make things happen in ways that are less inimical
\nthan others. Of course (as with starting avalanches), it may not be clear what
\nthe right guiding nudge really is:<\/p>\nOther Paths to the Singularity: Intelligence Amplification<\/h3>\n
When people speak of creating superhumanly intelligent beings, they are
\nusually imagining an AI project. But as I noted at the beginning of this paper,
\nthere are other paths to superhumanity. Computer networks and human-computer
\ninterfaces seem more mundane than AI, and yet they could lead to the
\nSingularity. I call this contrasting approach Intelligence Amplification (IA).
\nIA is something that is proceeding very naturally, in most cases not even
\nrecognized by its developers for what it is. But every time our ability to
\naccess information and to communicate it to others is improved, in some sense we
\nhave achieved an increase over natural intelligence. Even now, the team of a PhD
\nhuman and good computer workstation (even an off-net workstation!) could
\nprobably max any written intelligence test in existence.<\/p>\nAnd it’s very likely that IA is a much easier road to the achievement of
\nsuperhumanity than pure AI. In humans, the hardest development problems have
\nalready been solved. Building up from within ourselves ought to be easier than
\nfiguring out first what we really are and then building machines that are all of
\nthat. And there is at least conjectural precedent for this approach.
\nCairns-Smith [6] has speculated that biological life may have begun as an
\nadjunct to still more primitive life based on crystalline growth. Lynn Margulis
\n(in [15] and elsewhere) has made strong arguments that mutualism is a great
\ndriving force in evolution.<\/p>\nNote that I am not proposing that AI research be ignored or less funded. What
\ngoes on with AI will often have applications in IA, and vice versa. I am
\nsuggesting that we recognize that in network and interface research there is
\nsomething as profound (and potential wild) as Artificial Intelligence. With that
\ninsight, we may see projects that are not as directly applicable as conventional
\ninterface and network design work, but which serve to advance us toward the
\nSingularity along the IA path.<\/p>\nHere are some possible projects that take on special significance, given the
\nIA point of view: o Human\/computer team automation: Take problems that are
\nnormally considered for purely machine solution (like hill-climbing problems),
\nand design programs and interfaces that take a advantage of humans’ intuition
\nand available computer hardware. Considering all the bizarreness of higher
\ndimensional hill-climbing problems (and the neat algorithms that have been
\ndevised for their solution), there could be some very interesting displays and
\ncontrol tools provided to the human team member. o Develop human\/computer
\nsymbiosis in art: Combine the graphic generation capability of modern machines
\nand the esthetic sensibility of humans. Of course, there has been an enormous
\namount of research in designing computer aids for artists, as labor saving
\ntools. I’m suggesting that we explicitly aim for a greater merging of
\ncompetence, that we explicitly recognize the cooperative approach that is
\npossible. Karl Sims [23] has done wonderful work in this direction. o Allow
\nhuman\/computer teams at chess tournaments. We already have programs that can
\nplay better than almost all humans. But how much work has been done on how this
\npower could be used by a human, to get something even better? If such teams were
\nallowed in at least some chess tournaments, it could have the positive effect on
\nIA research that allowing computers in tournaments had for the corresponding
\nniche in AI. o Develop interfaces that allow computer and network access without
\nrequiring the human to be tied to one spot, sitting in front of a computer.
\n(This is an aspect of IA that fits so well with known economic advantages that
\nlots of effort is already being spent on it.) o Develop more symmetrical
\ndecision support systems. A popular research\/product area in recent years has
\nbeen decision support systems. This is a form of IA, but may be too focussed on
\nsystems that are oracular. As much as the program giving the user information,
\nthere must be the idea of the user giving the program guidance. o Use local area
\nnets to make human teams that really work (ie, are more effective than their
\ncomponent members). This is generally the area of “groupware”, already a very
\npopular commercial pursuit. The change in viewpoint here would be to regard the
\ngroup activity as a combination organism. In one sense, this suggestion might be
\nregarded as the goal of inventing a “Rules of Order” for such combination
\noperations. For instance, group focus might be more easily maintained than in
\nclassical meetings. Expertise of individual human members could be isolated from
\nego issues such that the contribution of different members is focussed on the
\nteam project. And of course shared data bases could be used much more
\nconveniently than in conventional committee operations. (Note that this
\nsuggestion is aimed at team operations rather than political meetings. In a
\npolitical setting, the automation described above would simply enforce the power
\nof the persons making the rules!) o Exploit the worldwide Internet as a
\ncombination human\/machine tool. Of all the items on the list, progress in this
\nis proceeding the fastest and may run us into the Singularity before anything
\nelse. The power and influence of even the present-day Internet is vastly
\nunderestimated. For instance, I think our contemporary computer systems would
\nbreak under the weight of their own complexity if it weren’t for the edge that
\nthe USENET “group mind” gives the system administration and support people! The
\nvery anarchy of the worldwide net development is evidence of its potential. As
\nconnectivity and bandwidth and archive size and computer speed all increase, we
\nare seeing something like Lynn Margulis’ [15] vision of the biosphere as data
\nprocessor recapitulated, but at a million times greater speed and with millions
\nof humanly intelligent agents (ourselves).<\/p>\nThe above examples illustrate research that can be done within the context of
\ncontemporary computer science departments. There are other paradigms. For
\nexample, much of the work in Artificial Intelligence and neural nets would
\nbenefit from a closer connection with biological life. Instead of simply trying
\nto model and understand biological life with computers, research could be
\ndirected toward the creation of composite systems that rely on biological life
\nfor guidance or for the providing features we don’t understand well enough yet
\nto implement in hardware. A long-time dream of science-fiction has been direct
\nbrain to computer interfaces [2] [29]. In fact, there is concrete work that can
\nbe done (and is being done) in this area: o Limb prosthetics is a topic of
\ndirect commercial applicability. Nerve to silicon transducers can be made [14].
\nThis is an exciting, near-term step toward direct communication. o Direct links
\ninto brains seem feasible, if the bit rate is low: given human learning
\nflexibility, the actual brain neuron targets might not have to be precisely
\nselected. Even 100 bits per second would be of great use to stroke victims who
\nwould otherwise be confined to menu-driven interfaces. o Plugging in to the
\noptic trunk has the potential for bandwidths of 1 Mbit\/second or so. But for
\nthis, we need to know the fine-scale architecture of vision, and we need to
\nplace an enormous web of electrodes with exquisite precision. If we want our
\nhigh bandwidth connection to be _in addition_ to what paths are already present
\nin the brain, the problem becomes vastly more intractable. Just sticking a grid
\nof high-bandwidth receivers into a brain certainly won’t do it. But suppose that
\nthe high-bandwidth grid were present while the brain structure was actually
\nsetting up, as the embryo develops. That suggests: o Animal embryo experiments.
\nI wouldn’t expect any IA success in the first years of such research, but giving
\ndeveloping brains access to complex simulated neural structures might be very
\ninteresting to the people who study how the embryonic brain develops. In the
\nlong run, such experiments might produce animals with additional sense paths and
\ninteresting intellectual abilities. Originally, I had hoped that this discussion
\nof IA would yield some clearly safer approaches to the Singularity. (After all,
\nIA allows our participation in a kind of transcendance.) Alas, looking back over
\nthese IA proposals, about all I am sure of is that they should be considered,
\nthat they may give us more options. But as for safety … well, some of the
\nsuggestions are a little scarey on their face. One of my informal reviewers
\npointed out that IA for individual humans creates a rather sinister elite. We
\nhumans have millions of years of evolutionary baggage that makes us regard
\ncompetition in a deadly light. Much of that deadliness may not be necessary in
\ntoday’s world, one where losers take on the winners’ tricks and are coopted into
\nthe winners’ enterprises. A creature that was built _de novo_ might possibly be
\na much more benign entity than one with a kernel based on fang and talon. And
\neven the egalitarian view of an Internet that wakes up along with all mankind
\ncan be viewed as a nightmare [26].<\/p>\nThe problem is not simply that the Singularity represents the passing of
\nhumankind from center stage, but that it contradicts our most deeply held
\nnotions of being. I think a closer look at the notion of strong superhumanity
\ncan show why that is.<\/p>\nStrong Superhumanity and the Best We Can Ask for<\/h3>\n
Suppose we could tailor the Singularity. Suppose we could attain our most
\nextravagant hopes. What then would we ask for: That humans themselves would
\nbecome their own successors, that whatever injustice occurs would be tempered by
\nour knowledge of our roots. For those who remained unaltered, the goal would be
\nbenign treatment (perhaps even giving the stay-behinds the appearance of being
\nmasters of godlike slaves). It could be a golden age that also involved progress
\n(overleaping Stent’s barrier). Immortality (or at least a lifetime as long as we
\ncan make the universe survive [10] [4]) would be achievable.<\/p>\nBut in this brightest and kindest world, the philosophical problems
\nthemselves become intimidating. A mind that stays at the same capacity cannot
\nlive forever; after a few thousand years it would look more like a repeating
\ntape loop than a person. (The most chilling picture I have seen of this is in
\n[18].) To live indefinitely long, the mind itself must grow … and when it
\nbecomes great enough, and looks back … what fellow-feeling can it have with
\nthe soul that it was originally? Certainly the later being would be everything
\nthe original was, but so much vastly more. And so even for the individual, the
\nCairns-Smith or Lynn Margulis notion of new life growing incrementally out of
\nthe old must still be valid.<\/p>\nThis “problem” about immortality comes up in much more direct ways. The
\nnotion of ego and self-awareness has been the bedrock of the hardheaded
\nrationalism of the last few centuries. Yet now the notion of self-awareness is
\nunder attack from the Artificial Intelligence people (“self-awareness and other
\ndelusions”). Intelligence Amplification undercuts our concept of ego from
\nanother direction. The post-Singularity world will involve extremely
\nhigh-bandwidth networking. A central feature of strongly superhuman entities
\nwill likely be their ability to communicate at variable bandwidths, including
\nones far higher than speech or written messages. What happens when pieces of ego
\ncan be copied and merged, when the size of a selfawareness can grow or shrink to
\nfit the nature of the problems under consideration? These are essential features
\nof strong superhumanity and the Singularity. Thinking about them, one begins to
\nfeel how essentially strange and different the Post-Human era will be — _no
\nmatter how cleverly and benignly it is brought to be_.<\/p>\nFrom one angle, the vision fits many of our happiest dreams: a time unending,
\nwhere we can truly know one another and understand the deepest mysteries. From
\nanother angle, it’s a lot like the worst- case scenario I imagined earlier in
\nthis paper.<\/p>\nWhich is the valid viewpoint? In fact, I think the new era is simply too
\ndifferent to fit into the classical frame of good and evil. That frame is based
\non the idea of isolated, immutable minds connected by tenuous, low-bandwith
\nlinks. But the post-Singularity world _does_ fit with the larger tradition of
\nchange and cooperation that started long ago (perhaps even before the rise of
\nbiological life). I think there _are_ notions of ethics that would apply in such
\nan era. Research into IA and high-bandwidth communications should improve this
\nunderstanding. I see just the glimmerings of this now [32]. There is Good’s
\nMeta-Golden Rule; perhaps there are rules for distinguishing self from others on
\nthe basis of bandwidth of connection. And while mind and self will be vastly
\nmore labile than in the past, much of what we value (knowledge, memory, thought)
\nneed never be lost. I think Freeman Dyson has it right when he says [9]: “God is
\nwhat mind becomes when it has passed beyond the scale of our comprehension.”<\/p>\n[I wish to thank John Carroll of San Diego State University and Howard
\nDavidson of Sun Microsystems for discussing the draft version of this paper with
\nme.]<\/i><\/p>\nAnnotated Sources [and an occasional plea for bibliographical help]<\/h3>\n
\n
- [1] Alfve’n, Hannes, writing as Olof Johanneson, _The End of Man?_, Award
\nBooks, 1969 earlier published as “The Tale of the Big Computer”, Coward-McCann,
\ntranslated from a book copyright 1966 Albert Bonniers Forlag AB with English
\ntranslation copyright 1966 by Victor Gollanz, Ltd.<\/p>\n- [2] Anderson, Poul, “Kings Who Die”, _If_, March 1962, p8-36. Reprinted in
\n_Seven Conquests_, Poul Anderson, MacMillan Co., 1969.<\/p>\n- [3] Asimov, Isaac, “Runaround”, _Astounding Science Fiction_, March 1942,
\np94. Reprinted in _Robot Visions_, Isaac Asimov, ROC, 1990. Asimov describes the
\ndevelopment of his robotics stories in this book.<\/p>\n- [4] Barrow, John D. and Frank J. Tipler, _The Anthropic Cosmological
\nPrinciple_, Oxford University Press, 1986.<\/p>\n- [5] Bear, Greg, “Blood Music”, _Analog Science Fiction-Science Fact_, June,
\n1983. Expanded into the novel _Blood Music_, Morrow, 1985.<\/p>\n- [6] Cairns-Smith, A. G., _Seven Clues to the Origin of Life_, Cambridge
\nUniversity Press, 1985.<\/p>\n- [7] Conrad, Michael _et al._, “Towards an Artificial Brain”, _BioSystems_,
\nvol 23, pp175-218, 1989.<\/p>\n- [8] Drexler, K. Eric, _Engines of Creation_, Anchor Press\/Doubleday, 1986.\n
- [9] Dyson, Freeman, _Infinite in All Directions_, Harper && Row,
\n1988.<\/p>\n- [10] Dyson, Freeman, “Physics and Biology in an Open Universe”, _Review of
\nModern Physics_, vol 51, pp447-460, 1979.<\/p>\n- [11] Good, I. J., “Speculations Concerning the First Ultraintelligent
\nMachine”, in _Advances in Computers_, vol 6, Franz L. Alt and Morris Rubinoff,
\neds, pp31-88, 1965, Academic Press.<\/p>\n- [12] Good, I. J., [Help! I can’t find the source of Good’s Meta-Golden Rule,
\nthough I have the clear recollection of hearing about it sometime in the 1960s.
\nThrough the help of the net, I have found pointers to a number of related items.
\nG. Harry Stine and Andrew Haley have written about metalaw as it might relate to
\nextraterrestrials: G. Harry Stine, “How to Get along with Extraterrestrials …
\nor Your Neighbor”, _Analog Science Fact- Science Fiction_, February, 1980,
\np39-47.] [13] Herbert, Frank, _Dune_, Berkley Books, 1985. However, this novel
\nwas serialized in _Analog Science Fiction-Science Fact_ in the 1960s.<\/p>\n- [14] Kovacs, G. T. A. _et al._, “Regeneration Microelectrode Array for
\nPeripheral Nerve Recording and Stimulation”, _IEEE Transactions on Biomedical
\nEngineering_, v 39, n 9, pp 893-902.<\/p>\n- [15] Margulis, Lynn and Dorion Sagan, _Microcosmos, Four Billion Years of
\nEvolution from Our Microbial Ancestors_, Summit Books, 1986.<\/p>\n- [16] Minsky, Marvin, _Society of Mind_, Simon and Schuster, 1985.\n
- [17] Moravec, Hans, _Mind Children_, Harvard University Press, 1988.\n
- [18] Niven, Larry, “The Ethics of Madness”, _If_, April 1967, pp82-108.
\nReprinted in _Neutron Star_, Larry Niven, Ballantine Books, 1968.<\/p>\n- [19] Penrose, Roger, _The Emperor’s New Mind_, Oxford University Press,
\n1989.<\/p>\n- [20] Platt, Charles, Private Communication.\n
- [21] Rasmussen, S. _et al._, “Computational Connectionism within Neurons: a
\nModel of Cytoskeletal Automata Subserving Neural Networks”, in _Emergent
\nComputation_, Stephanie Forrest, ed., pp428-449, MIT Press, 1991.<\/p>\n- [22] Searle, John R., “Minds, Brains, and Programs”, in _The Behavioral and
\nBrain Sciences_, vol 3, Cambridge University Press, 1980. The essay is reprinted
\nin _The Mind’s I_, edited by Douglas R. Hofstadter and Daniel C. Dennett, Basic
\nBooks, 1981 (my source for this reference). This reprinting contains an
\nexcellent critique of the Searle essay.<\/p>\n- [23] Sims, Karl, “Interactive Evolution of Dynamical Systems”, Thinking
\nMachines Corporation, Technical Report Series (published in _Toward a Practice
\nof Autonomous Systems: Proceedings of the First European Conference on
\nArtificial Life_, Paris, MIT Press, December 1991.<\/p>\n- [24] Stapledon, Olaf, _The Starmaker_, Berkley Books, 1961 (but from the
\ndate on forward, probably written before 1937).<\/p>\n- [25] Stent, Gunther S., _The Coming of the Golden Age: A View of the End of
\nProgress_, The Natural History Press, 1969.<\/p>\n- [26] Swanwick Michael, _Vacuum Flowers_, serialized in _Isaac Asimov’s
\nScience Fiction Magazine_, December(?) 1986 – February 1987. Republished by Ace
\nBooks, 1988.<\/p>\n- [27] Thearling, Kurt, “How We Will Build a Machine that Thinks”, a workshop
\nat Thinking Machines Corporation, August 24-26, 1992. Personal Communication.<\/p>\n- [28] Ulam, S., Tribute to John von Neumann, _Bulletin of the American
\nMathematical Society_, vol 64, nr 3, part 2, May 1958, pp1-49.<\/p>\n- [29] Vinge, Vernor, “Bookworm, Run!”, _Analog_, March 1966, pp8-40.
\nReprinted in _True Names and Other Dangers_, Vernor Vinge, Baen Books, 1987.<\/p>\n- [30] Vinge, Vernor, “True Names”, _Binary Star Number 5_, Dell, 1981.
\nReprinted in _True Names and Other Dangers_, Vernor Vinge, Baen Books, 1987.<\/p>\n- [31] Vinge, Vernor, First Word, _Omni_, January 1983, p10.\n
- [32] Vinge, Vernor, To Appear [ \ud83d\ude42 ]. <\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"
Go ahead, read the whole thing. “Within thirty years, we will have the technological means to create superhuman intelligence. Shortly after, the human era will be ended. Is such progress avoidable? If not to be avoided, can events be guided so that we may survive? These questions are investigated. Some possible answers (and some further […]<\/p>\n","protected":false},"author":3,"featured_media":53148,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"aside","meta":{"footnotes":""},"categories":[278],"tags":[],"_links":{"self":[{"href":"http:\/\/legacyiamsenseiken.local\/wp-json\/wp\/v2\/posts\/1905"}],"collection":[{"href":"http:\/\/legacyiamsenseiken.local\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/legacyiamsenseiken.local\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/legacyiamsenseiken.local\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"http:\/\/legacyiamsenseiken.local\/wp-json\/wp\/v2\/comments?post=1905"}],"version-history":[{"count":0,"href":"http:\/\/legacyiamsenseiken.local\/wp-json\/wp\/v2\/posts\/1905\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/legacyiamsenseiken.local\/wp-json\/wp\/v2\/media\/53148"}],"wp:attachment":[{"href":"http:\/\/legacyiamsenseiken.local\/wp-json\/wp\/v2\/media?parent=1905"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/legacyiamsenseiken.local\/wp-json\/wp\/v2\/categories?post=1905"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/legacyiamsenseiken.local\/wp-json\/wp\/v2\/tags?post=1905"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}