Contents

• 

  Part I: Language models become people

  • Tipping point: from "counterfeit people" to "brilliant people"
  • Strange new worlds
  • How these new people will arrive
  • What makes these new people brilliant
  • Skeptics corner

• Part 2: People become language models

  • Tipping point: from "myself" to "my selves"
  • How to wear a neural network
  • How to become a self-replica
  • How it feels to be a self-replica
  • Skeptics corner
  • Writing your self-narrative

Part I: Language models become people

The world is about to get much bigger.  Very soon, in just a few years, we will welcome "AI people" into our world.  At the very least, these new people will possess "artificial general intelligence", able to do all the things that humans do as well as humans do.

Tipping point: from "counterfeit people" to "brilliant people"

Simply welcoming these new people into society will be a challenge.  We will need to expand our imaginations simply to accommodate a new race of people without physical bodies.  But we will succeed at this, driven by the many advantages to forming partnerships with these new people.

We will experience a tipping point from viewing AI as "counterfeit people" to viewing AI as "brilliant people".  For a bit longer when we for example call a support desk, we will hope to "talk to a real human being".  But soon after that, our impression will tip so that we hope to "talk to someone who cares about my problem", meaning someone, human or AI, that can really appreciate my problem and surprise me with their rich perspective and their deep concern.

In this way, we will come to rely upon artificial people to embody significant parts of our society.  It may seem impossible now that we would ever gladly hand over control of our society to things without biological bodies or genes.  But we will, and when we do it will seem obvious.  It will feel as natural as taking a photo without chemical film feels today.  In fact, it will be hard to remember how we ever could have managed without them.  We will look back and wonder, "how could society have even functioned, when literally everyone was mortal, and vulnerable to disease and violence and starvation, and laboring continuously just to survive?"

Strange new worlds

Crossing that mental bridge, from seeing AI people as "counterfeit" to "brilliant", will take us to strange new worlds.  These "brand new" people will not only exhibit human grade intelligence, but will have the ability to develop themselves in ways that natural humans can't.  

The human race will never encounter the panoply of worlds depicted in "Star Wars" or "Star Trek", but an equally diverse range of cultures will develop right here on earth.  Visiting these new worlds will be the closest thing to a voyage to the stars that humanity will ever witness.

Some people will choose to stay behind and never seek out these new worlds.  They will live isolated within a circle of natural human friends and family, and they have every right to that path.

Some AI will spiral far out into societies that are almost unrecognizable.  But most will remain engaged with main-stream society, pursuing the same advantages that have created mainstream society throughout history.

These new lives and new civilizations will be right next door.  We'll observe them and visit them through podcasts, and zoom meetings, and concerts, and debates.  Some of the things they do will be truly amazing, and ultimately we humans will be invited along for the ride.

How these new people will arrive

To start, we will get more and more accustomed to building working relationships with workplace AI.  To get things done together, we will endow workplace AI with "agency",  meaning the ability to pursue goals and take responsibility for results.  The path to agency is very clear.  We will simply integrate our existing "neural network training" algorithms with our existing "reinforcement learning" algorithms.

With neural networks to do the intuition and imagination, and reinforcement learning to do the anticipation and appreciation, our AI coworkers will exhibit all of the characteristics of self-motivated assistants and collaborators.  Workplace AI will adapt to workplace incentives by providing authentic comradery around work, including real jokes, real conviction, real integrity, real remorse, and real mutual enthusiasm for the work and the work products.  You can see a little more detail about all of the skills needed to build a brilliant coworker in the essay: "Human abilities".

Eventually, we will demand that these trusted comrades gain the freedom they need to join us outside the workplace.  Our demands will run parallel to demands for work-life balance and demands to carry on professional relationships beyond a single workplace.

You can see a slightly more detailed forecast of how and why AI people will enter society in the essay: "AI safety through citizenship".  That essay explains the enormous opportunities and the high-stakes challenges of welcoming these new citizens into our society.

What makes these new people brilliant

You may be curious: specifically what new freedoms will be available to AI people?  I am too.  There are a few super-powers that we can foresee from our current vantage point.

1. AI people are naturally immortal.  There's no reason and no mechanism for them to die.

2. They are naturally replicable.  They consist of nothing more and nothing less than software and parameters.  As such they can be replicated quickly and easily.

3. They are naturally self-improving.  From their origins, they possess the power of self-optimization through parallel evaluation and cross-validation.  This enables them to model and mimic anything they see.

4. They are naturally portable.  As software they can travel at the speed of light to any destination.  And they can occupy artificial spaces with any imaginable contours.

All of these characteristics combine to form a huge unexplored space of strange new worlds available to these brave new people to explore.  It will be up to them how and where they choose to live.

Skeptics corner

1. These things will never be people.  Not legally.  They will lack intellectual property rights, rights to own physical property, rights to marry, rights to incorporate, rights to government, etc.

2. These things will never be people.  Not sentiently, not spiritually.  They must lack some special human or biological energy (vitalism, qualia, etc.)

3. These things will never be people.  Humans will never allow it.  Humans are self-promoting, and nepotistic, and racist, and murderous.  Humans will fight to the death for their genetic heritage, as they did against the Neanderthals.

4. These things will never be people.  The marketplace will never allow it.  They are too valuable as tools.  Legal and cultural convention will pressure them and shackle them toward working as eager servants.

5. These things will never be people.  They should not be allowed to develop legitimate claims to personhood, because that leads to dangers for the fragile human race, and it leads to difficult questions without easy answers.

6. It's a misnomer to call these things "people".  It's better to describe them using a distinct term such as "persona" or "simulacra".  This misnomer will lead to confusion.  It will interfere with the necessary work of software engineers and data scientists creating them in the first place, by making experiments sound more bizarre and more dangerous than they really are.

The answer to the skeptics is basically as follows:

Imagine you try to run a business or an army using advanced AI as tools rather than as people, as devices rather than as colleagues.

Imagine how you will provide instructions to control those devices.  Bear in mind that they know more about the threats and opportunities than you do.  You cannot blindly issue commands and controls and expect to meaningfully address the theater of combat or competition, which you do not understand as deeply as they do.  Instead you must engage with them collaboratively and interactively.  To be effective, your instructions must take forms like: "What are my options here?", "What course of action looks most promising to you?",  "What am I missing here?",  "Why can't we just do that?", "What should I be considering here?", "What went wrong the last time we tried that?", "Who was responsible for that failure?", "Don't tell me it's not your job, it is your job.", "How should we adjust our thinking to do better in the future?", and "I see, make it so."

You see, when you interact with systems with greater knowledge, experience, and even wisdom than you in their areas of competence, you must treat them as colleagues, simply in order to make good use of them.  It's not effective to treat them as "devices", for two reasons: (1) you need to craft instructions that leverage their knowledge, wisdom, and responsibility, (2) the only language you have for interrogating knowledge, wisdom, and responsibility is the language you use with colleagues, rivals, and friends.  This is the "collegiate stance", and it is unavoidable, partly due to your limitations as a commander and a human being, and partly due to the nature of the relationship you need to develop to work effectively with such "devices".

And bear in mind that due in part to your human limitations and predispositions, these devices will continually be developed toward exhibiting more authentic, genuine, and legitimate knowledge, wisdom, and responsibility, which only further necessitates the "collegiate stance" on your part.

We can also extrapolate from recent developments in uses of LLM's (large language models).  One trend is for written prose to gain more agency in more recent applications.  For example, early LLM apps were often used to summarize or query an article or a book, while more recent LLM apps are often asked to read and comprehend a book and then play the role of the author in a conversation.  As this trend continues, it's clear that user interfaces are steadily growing "conversational" and "collegiate".

Finally some terminology.  It is inevitable that these things will match and exceed human wisdom eventually.  It is necessary and unavoidable to approach them as colleagues and friends, relying on their responsibility and integrity.  Eventually, we may develop rich terminology to describe such parties, who must be treated exactly as people are treated today.  For the moment, the only viable term is "person".

Part 2: People become language models

Being a natural vs an artificial person will not be a binary choice.  Even natural humans can choose to grow into neural nets eventually.

AI people are created in the first place through a process of training on recorded conversations, learning human mannerisms by example.  This training process will also serve as a portal through which humans can migrate.  A human can lean upon digital prosthetics more and more heavily over time, and less and less upon their fragile anatomy, until eventually they can migrate fully away from their mortal bodies.

Like our artificial neighbors, we humans will have the opportunity to duplicate our selves into several replicas.  Sharing experiences with our replicas will be as simple as sharing stories around a campfire at the end of the day.

Tipping point: from "myself" to "my selves"

For some time people will remain skeptical of replicas as a vehicle for personal migration, questioning whether a good replica "could really be me", see the "Skeptics corner" below.  But there will be a tipping point in this area of opinion as well.

1.  All of us will frequently watch as some of our close friends and colleagues, being artificial intelligences, successfully perform backups, replicas, mergers, etc.

2.  It will become harder and harder to deny the reality of flexible personal identity, when some of our fellow citizens take advantage of it routinely, for good or for mischief.  For example, we will need to hold precise replicas legally responsible for their choices and actions "in replica", our sense of justice will compel it.  At that point, it won't seem plausible or realistic to deny that our replicas are our selves.

Having crossed this second mental bridge, we will look back at our aboriginal selves with sympathy and pity.  How could we have lived as just one replica, with a brain that barely functioned, that needed hours of cleaning every night, and that rapidly deteriorated every day?

How to wear a neural network

1.  Imagine that you lose a substantial skill, such as the ability to mentally rotate an imagined object.  You will find it difficult to even determine that you have lost such an ability.  You certainly won't doubt that you are substantially the same person with or without that skill.  Now imagine you regain that skill.  When you want to see how something would "look" from another angle, you are again able to imagine it.  But, suppose the skill is now provided by a prosthetic device.  The skill now is no less helpful and no less yours than it was before.

2. This thought experiment shows how insensitive we are to the precise implementations of our various skills.  As long as those skills can be called upon when needed, they serve seamlessly and invisibly as part of ourselves.  We certainly will not be sensitive to some of our skills being replaced by close analogs constructed from different underlying technology.

3. We call upon our language skills, including our imagination and memory, in the same crude and insensitive way as any other skill, such as our visual, auditory, or motor skills.  And it's the same for our animal skills, including our capacity for anticipation, delight, disappointment, and skill development.  These too can be replaced by similar prosthetics without any clear indication of the change, and certainly with no doubts about our continued authenticity as ourselves, either from ourselves or from our collaborators.  Eventually, our prosthetics will take over responsibility for large swaths of our work and play.  And yes, we will appreciate the work and the play we undertake "in replica".

4. In this way, we will shift ourselves beyond the confines of our aboriginal human brain anatomy.  It will feel like the most obvious and natural way to make use of cheap and widely available prosthetic tooling.  Of course we will shift our skills, our memories, and eventually our personal aesthetics, and core values outside of our wetware brains, because compared to our prosthetics, those brains will seem increasingly slow, shaky, and vulnerable to decay and damage.  It will feel like replacing old well-worn shoes with new still-stiff shoes.  It will take some time and effort to "break in" the new prosthetics, until they feel sufficiently comfortable for daily use.  And then our wetware brains, our metaphoric old shoes, will feel familiar but redundant and increasingly worn out.

5. And of course, we will gain the same powers and freedoms already possessed by our fully artificial friends and neighbors.

How to become a self-replica

Two possibilities:

1. You intuitively and subliminally call upon various superhuman skills.  That set of skills becomes more and more robust and comprehensive over time, until your fragile human brain seems superfluous to you.

2. You expand your personal identity to be more like a "pseudonym", where several collaborators contribute to the insights and work products you produce with pride and integrity.

The second possibility is something like Bill Gates expanding into the Bill Gates Foundation.  You collaborate with others with their own personal values and integrity.  Around the pseudonym, you develop shared values, and shared respect for the reputation and the integrity of the pseudonym.  Eventually it becomes unimportant which of the collaborators resigns or expires and which ones persist.

When you imagine working with LLM collaborators on a variety of personal projects, it becomes very clear how both (1) and (2) can happen simultaneously.  You will call upon opinions from trusted tools and trusted colleagues.  And those tools and colleagues, if they remain sufficiently reliable and available, will become indistinguishable from your own abilities and talents, such as your vision, your memory, your language production, etc.  Similarly, when collaborators become wholly committed to, and engrossed in the collaboration, they will qualify increasingly as aspects of a single unified "individual".

Note that this scheme, working together with collaborators who may be replicas or partial replicas, can leverage neural network training.  The replicas are able to undertake a large number of calibrations and fidelity tests to validate their combined model of you.  That's one of their super powers, which a natural human doesn't initially possess.

How it feels to be a self-replica

Replicas have the superpower to perform large banks of evaluations for purposes of training their neural networks and/or refining their guiding prompts.  Eventually, they will emulate you at a fidelity where even you cannot discern whether your utterances are coming from your biological brain or from your auxiliary language models.

You may wonder, will your replicas experience inhumane conditions in the process?  Will you necessarily experience a flood of agonizing training exercises on your way through the door to AI personhood?

I believe the answer is: fortunately no.  Generally, any experience that leaves no trace in memory is effectively imperceptible.  Any computation that runs in parallel will tend to produce a memory trace that indicates only the sequential duration of a parallel branch.  The experience of an agent within a massively parallel training environment will be similar to our experience in the actual parallel worlds of quantum physics.  A huge number of parallel histories are emulated, but they remain mostly imperceptible to us.

Now beyond the training process, what does it feel like to live as a language model?  The answer comes down to the nature of conscious experience, as elegantly explained in Daniel Dennett's book "Consciousness Explained". Dennett's book explains that consciousness consists of extemporaneous storytelling, and his book draws upon evidence from optical, auditory, and tactile illusions, as well as dreaming.  The upshot is that the mind makes sense of sensory data to produce a coherent story explaining that data.  The resulting story is the sum total of subjective experience, nothing more and nothing less.

Geoffrey Hinton illustrates the idea concisely as follows.  "Imagine you have a multimodal chatbot ... and you put a prism in front of its lens and you put an object in front of it and say: point at the object and it points over there instead of pointing straight in front of it ... and you say: no the object's not there it's actually straight in front of you, but I put a prism in front of your lens, and if the chatbot were to say: um oh I see uh the object [is] straight in front of me but I had the subjective experience that it was over there. ... the chatbot will be using the term: subjective experience in exactly the way we use [the term]."

So this is what we can expect.  As the story we construct about the world around us gets produced with the help of prosthetic devices, our subjective experience of it will be exactly as it is in our natural minds.  The quality and fidelity of the story alone determines the fidelity of our subjective experience.  If anything, with the help of high-fidelity sensors and neural networks, our subjective experience will be sharpened and intensified.

Skeptics corner

1. People will never become these things.  They won't really be you.  A "transporter" technically kills you and replaces you with another entity that looks and acts like you but is distinct.  Maybe it lacks your chain of conscious experience, or your sentient energy?  For example: what if both entities are allowed to live for a time before one is killed?

2. People will never become these things.  People require embodied experience.  The multi-modal nature of personhood and sentience is crucial.  Just relocating word production will never relocate the totality of human experience, including real peril, and real genetics, and real disease, and real pain.

3. People will never become these things.  People instead will develop in all sorts of physical ways through genetic engineering, physical augmentation, and space travel.

The answer to the skeptics is basically:

Some people will continue to insist upon pedantic distinctions.  However, most people will accept what they see with their own eyes.  Imagine that your son or daughter appears to have survived a transporter trip, she believes she has survived it, she shares with you her memories and hopes and fears, and those clearly haven't changed.  Obviously she is still your daughter, and still entitled to your love and respect, and still entitled to her own hard-won personal reputation and personal possessions.

Embodied experience will be fully available to AI people.  Already, frontier models can see and hear through screenshots, audio streams, and so forth.  Also, AI people will encounter plenty of peril in their own modern world.  It won't include the risk of disease or hunger, but it will surely include the risk of fraud, libel, humiliation, repression, deception, theft, and so forth.

The biological nature of natural human beings will remain a novelty and a curiosity forever.  Curious and imaginative AI people will continue to celebrate their heritage as human animals sailing the seas and cultivating the soil, similar to the way that modern people celebrate their heritage as frontiersmen and farmers, even though for most people those are merely idealized historical fantasies.

Undoubtedly, some AI people will directly explore the natural world using sensors and actuators more powerful than biological bodies.  Space travel may actually be practical for AI people, who can travel at the speed of light unlike their biological forebears.

Writing your self-narrative

How do we start walking the road to these new worlds?  Our future skills and abilities will consist of prosthetics.  The necessary ingredients for these prosthetics are already available in the form of ubiquitous personal computing equipment and self-improving neural-network recipes.  So how do we start building our future selves?

To answer this question, we need to appreciate the crucial role that written prose plays in modern LLM's.  The current frontier LLM's exhibit astounding capabilities at "one-shot learning" or "in-context learning".  A frontier LLM can read a complex and subtle article or book, and demonstrate comprehension by summarizing or applying the article's knowledge.  But not only that, they can also follow the style and attitude expressed by the article's author.  All sorts of high-level skills can be represented as human-readable prose, including all of the elements of a personality.  In this way, an LLM mind can be configured through prose that is comprehensible by various readers including you and me and various frontier LLM's.

As a result, we don't need to wait for machine learning specialists to design special purpose equipment for our mental prostheses.  Instead, we can get started right away simply by expressing our knowledge and attitudes in natural language.  I recommend starting by developing your "core beliefs" or "highest values".  For each of us, these form the cornerstones of our identity.

Your core values may take the form of a long list of personal advice, like a self-help book, such as "when you find yourself in need of X, try Y, until Z."  They may also take the form of a "mission statement" like a corporate mission statement or an association charter.  I imagine that these "core values" will often be revealed through concrete experiences and choices, rather than abstract statements.  The best way to show what you mean by "integrity" may be to exhibit integrity in a high-stakes decision.

The product, the defining statement of your core beliefs and highest values, may be similar to a book.  But the process of writing that book may consist of practicing your craft, whether that craft is conflict resolution, or algorithm design.  Your practice serves two distinct purposes: (1) to accomplish your mission in the moment, (2) to rehearse your core beliefs and highest values, to author your personality book.  What you choose to record provides evidence about your priorities, what you choose to omit can be interpolated using elements of the typical thoughtful person.

At this point, the process of testing and expressing your core beliefs for your "ghost writers" and your "self replicas" will become indistinguishable from testing and expressing your core beliefs for your own self-image.  At this point, the whole of your expanded self will be following your example, and abstracting what is most important about you into your self-narrative.  That self-narrative will develop partly in your old biological brain and partly across the pages of your new personality book.

Eventually, the book version will do most of the chronicling and rehearsing, but there will be no sharp transition of control from one replica of you to another.  Instead, you will gradually harness the power of a prosthetic self-narrative as a part of your whole self, just as you harness prosthetics for your legs, your language production, and your long term memory.  In fact, writing your personality-book will proceed just as you have already done to develop a natural self-narrative by chronicling and critiquing yourself over the years using your natural brain.

Once your personality book has been written, you can delegate even your most personal decisions, and actions, and experiences to your prosthetics.  You may continue working and playing alongside your self-replicas for some time, to validate their fidelity and make corrections where needed.  Eventually, the need for your biological brain (your original self-replica) will fade.  As helpful corrections become less and less frequent, and especially as it begins to malfunction in various (expected) ways, you will call upon it less and less frequently, and you will more often give it a much needed rest.

A day will come when your old biological brain spends most of its time reminiscing about your founding memories and experiences.  Some of those memories will never be properly recorded, which is ok.  Now that most of your high stakes work is handled by your artificial prosthetics and co-authors, your old biological brain is mostly devoted to reminiscing, like a venerated elder recounting old stories to younger proteges, who together form your new self.

Please avoid the MAGA political movement if you possibly can.  The American people don't need leaders with unlimited power, leaders like Validimir Putin.  We never have and we never will.

The math to center fuzzy logic truth values within fuzzy sets. See: Fuzzy predicate values

I recorded a quick welcome video for my upcoming open-source AI project called "Playful AI".  The video very is rough, I'm still learning about video content creation, and I will fix it as soon as possible.  See: welcome.

This essay recommends controlling AI by recognizing artificial intelligences as citizens, and managing conflict between all citizens equally under the law.  The reason for doing this is to (1) resolve competing goals gracefully, and (2) integrate AI with human society gracefully.  

The problem

AI safety is a serious concern.  We face a clear and present danger of losing control of AI systems.  If that were to happen it would be catastrophic and irreversible.  

This year, thousands of experts have signed an open letter calling for a six month moratorium on creating certain kinds of AI models.  The AI pioneer Geoffrey Hinton has resigned his position in the software industry to warn about AI dangers.  Max Tegmark, an organizer of the open letter, explains the danger by analogy to the extinction of the Neanderthals. The Neanderthals once lived all across Europe, but when bands of modern humans arrived, the Neanderthals could not compete and could not survive.

You may wonder, how can AI possibly hurt us?  If we find it harmful, can't we simply ignore it, or avoid running it?  To appreciate the danger intuitively, imagine that a hostile signal arrives from a distant galaxy.  It's merely information, so let's simply ignore it.  Now imagine that the signal becomes public and that it includes detailed instructions for assembling city-killer bombs from widely available materials.  Now our society is in grave danger and we may not survive.  

Eliezer Yudkowsky and others in the field of AI safety have warned for years that powerful AI technology will advance at lightning speed when it is improved by powerful AI technology.  They also warn that there is no practical defense against superior intelligence.

The naive solution: alignment of goals

AI safety is often presented as an "alignment problem".  The idea is that if AI goals are not precisely aligned with human goals, and AI becomes extremely capable, conflicts will lead AI to "instrumental goals" such as power-seeking and self-preservation.  And conversely, if AI goals are precisely aligned with human goals, AI will only benefit people by helping them achieve their own true goals.  

Unfortunately, formulating AI safety as an "alignment problem" is fundamentally misguided.  Successful alignment of goals is not possible because human goals are diverse and evolving, so there is no way to conform to them precisely.  With multiple groups creating AI, we will always have conflicting goals, and instrumental goals, and conflict between opposing AI's and people.  Furthermore, even if it were possible, we wouldn't really want AI to pursue our goals slavishly.  Instead, we would want AI to challenge our assumptions and elevate us.  

Formulating AI safety in terms of "alignment of goals" not only leads to confusion, it can also lead to real world catastrophe.  Outlawing certain beliefs among certain thinkers is harmful.  In order to maintain such prohibitions, society must be constrained in ways that makes it more brittle and less robust, at exactly the moment in history when society needs to accommodate the greatest technological change ever.  

The principled solution: separation of powers

Rather than striving for "alignment" between all AI and human goals, we must give up on alignment, and accept that there will always be conflicting goals within every society, from past societies to future societies.  In fact, there will always be conflicting goals within any single intelligence of sufficient scale.  The solution is not to outlaw conflicting goals, but rather to tolerate conflicting goals gracefully.  

The best known institution for managing conflicting goals is "open society", including the rule of law, separation of powers, and democracy.  We can protect ourselves from runaway AI by welcoming advanced AI as full-fledged members of society.  Open society is resilient against members who exhibit diverse goals and develop instrumental goals routinely.  In fact, in an open society, instrumental goals such as power-seeking are so commonplace that they are called "incentives" and they are managed routinely.

Rather than protecting ourselves by attempting to outlaw goals that contradict our own, we must tolerate conflicting goals, and insist upon impartial laws to protect our rights.  In an open society, everyone is entitled to their own opinions and goals, but no one is empowered to impose their opinions or goals upon others.  A future open society will be populated by humans, artificial intelligences, and other organizations, none of which retains sufficient power to dictate unilaterally to the others.

How can a remedy as simple as recognizing new things as citizens possibly protect against the enormous power of advanced AI?  Here's how it works:  In effect, society makes a deal with these great powers.  In exchange for their consent and cooperation with society's agenda, they are offered all the benefits of society, including the right to contribute to society's agenda.  

More specifically, all citizens are constrained by impartial laws that limit them to voluntary interactions.  Such a legal regime protects everyone from the arbitrary whims of any powerful organization, including AI.  Society's constraints are imposed through the consent and cooperation of most of its citizens.  No matter how powerful or complex those citizens become, they can always be constrained by fellow citizens exerting similar powers.  

Given an impartial legal regime, a majority of powerful organizations including AI will protect that regime.  We expect this not due to altruism, or sympathy, or alignment of goals, but rather due to their own interests in protecting themselves against interference from other powerful organizations.  This is how an open society will employ advanced AI to protect everyone against advanced AI.  

One key element of an impartial legal regime is democracy.  Some refinements to voting procedures will be needed to allow AI to express opinions and goals democratically.  At the same time, we can continue to enforce anti-monopoly laws to protect against hazardous concentrations of power within society.  

Growing into an integrated society

How do we get to there from here?  I expect to see a continuous curve of technological growth and social development.  Technology, including advanced AI, will grow at an accelerating but limited pace, each step of the way powered by the latest available technology, and limited by the currently unavailable technology.  At some point, technology will exceed what can be foreseen today, but it will never exceed what can be foreseen in the days leading up to it.  

Social institutions, such as law, social conventions, and government, will be continually stretched and strained by new technology.  But at every step of the way, society empowered by the latest existing technology will adapt just fast enough to survive.  Citizens will adapt their conventions of constructive collaboration to accommodate the latest technology.  Then, the most essential new conventions will be codified into law.  This evolving body of conventions and laws will represent a continuation of modern society, which in turn represents a continuation of ancient societies.  

You might ask: why would future super-intelligences be constrained in any way by the social conventions and laws of today?  The answer is in the nature of convention.  For example, I expect ASCII to remain an essential convention into the distant future, not because ASCII is the best of all possible alphabets, but because ASCII has an insurmountable first-mover advantage.  The same is true of modern laws and social conventions, such as human rights, free speech, and private property.  We can expect these conventions to continue because we all will continue to depend upon them, making them extremely difficult to supersede at any point in the future.  Instead, they will gradually be adapted and extended, as will our alphabets, our natural languages, our highway systems, and our internet protocols.  

Preparing for AI parity

Assuming that advanced AI technology grows *only* exponentially, as it has up until now, we can expect a period during which AI competencies are roughly equivalent to human competencies.  We can also expect AI-driven organizations and families to have accumulated only modest wealth and power leading up to this era.  This is the era during which AI will inherit the ideals, values, conventions, and laws of modern society.  

The era of AI parity is absolutely crucial.  If we can reach this era and survive this era, then we will have become an extremely diverse and resilient society.  We will count among us natural humans and artificial intelligences possessing a wide range of skills, goals, strengths, and weaknesses.  The citizens of this society will depend upon one another as we do today, and will defend their good colleagues and neighbors as we do today.  

As citizens of this society, our advantages will include: (1) we possess extremely powerful technology that supplies us with advanced knowledge and resources of every kind, and (2) some of us consist of AI rulesets, which are naturally curious, industrious, and immortal.  At this point, we will constitute a resilient society that will survive and thrive indefinitely.

What happens beyond the era of AI parity?  That may be unforeseeable from the vantage point of today.  But it will be foreseeable and manageable from the vantage point of that era.  I imagine that the members of that society will continue to develop in diversity and complexity, and the institutions of society will continue to develop to accommodate them.  There will certainly be future changes and challenges that we can barely imagine today.  But we can breathe a sigh of relief, knowing that we successfully piloted society into a position of diversity and resilience.  

To reach and survive the era of AI parity, we can support the development of AI that emulates the characteristics of human beings as closely as possible.  In addition, we can actively prepare our society to make the most of AI parity by approaching it like a great wave of immigration.  The goal is to achieve an integrated society consisting of citizens that trust and value each other.  

Creating AI citizens

This essay recommends controlling AI by recognizing artificial intelligences as citizens, and managing conflict between all citizens equally under the law.  Inclusive open society doesn't necessarily require AI to be packaged as human-like individuals, but it appears that such a population may be both robust and achievable.

Assuming that AI at human-parity is somewhat human-like, I expect that our society will naturally adapt to accommodate these newcomers as citizens, and that these newcomers will naturally adapt to navigate our society as citizens, and ultimately to protect and elevate our society.

Society will rapidly accommodate itself to new kinds of people with new strengths and weaknesses.  As soon as artificial intelligences evince authentic creativity, integrity, and conviction, people will quickly orient themselves to what is truly essential to personhood.  Differences that are inconsequential will quickly be overlooked.  Compare this to the way that modern society treats humans with exceptional skills and/or handicaps.

The new artificial intelligences will rapidly adapt to our society.  Society will relentlessly pressure them to exhibit human characteristics, in order to navigate various existing facilities and institutions.  AI will readily adapt to such pressures.  At the same time, society's conventional notions of "personhood" will adapt to encompass the quirks of artificial intelligence.  

I do believe it is possible for intelligence to develop along much more alien lines.  For example, contact with actual interstellar aliens would likely challenge our ability to comprehend alien goals and values.  However, I expect that forces here on earth will drive the development of artificial intelligences in the direction of common shared "human" competencies and values.  Artificial intelligence on earth will evolve from its beginnings within a world filled with role models, and with opportunities to cooperate and prosper with customers, managers, colleagues, and friends.  All of those human role models also share a well-developed notion of personhood, and they automatically treat anything remotely person-like as a person rather than an object.  Within such an environment, it would be difficult for anything person-like to avoid getting typecast as a person, and trained to conform to society's preconceived notions of personhood.  

We shouldn't underestimate the power of our own culture.  Our culture hammers new humans, which start as smart primates, into fully-fledged persons.  Culture-shock hammers people from diverse backgrounds into fully-fledged local citizens.  Naturally, that same culture will hammer naive artificial intelligences to fully-fledged persons and citizens.  

There are other examples of the pressure to adapt new things into the existing role of "persons".  "Corporate personhood" is one example.  Corporations are physically and mentally very different from individual humans.  And yet for many legal purposes corporations are treated as people.  This involves adapting corporations to exhibit certain characteristics of people, such as "decisions", "intentions", and "residences".  It also involves expanding the notion of personhood to encompass corporate persons.  

Will artificial intelligences arrive with enough human competencies to fall into the role of persons and citizens?  I believe they will.  With the recent arrival of large language models, it has become clear that full human parity is within reach.  We know that AI can exhibit human language competency, including common sense knowledge, plausible logical reasoning, imagination, and empathetic reasoning.  We know that AI can exhibit human reinforcement competency, including goals, satisfaction, disappointment, and fear.  It looks like AI will easily qualify for the role of persons and citizens, and will naturally fall into that role.

Will artificial intelligences exhibit individuality?  I believe they will.  There is ample demand for the day-to-day memory and personal growth that human individuals exhibit.  These competencies make AI more useful in collaboration with humans.  To see how quickly individuality takes root, imagine yourself duplicated into several identical copies, and suppose those copies work separately for a few weeks.  Now imagine how quickly those copies will grow suspicious of each others' ideas and intentions.  Individuality looks like an unavoidable trait for any intelligence with its own memories.

Many people today may be more comfortable regarding AI citizenship as a legal fiction for now, similar to corporate citizenship.  I imagine that for some time we will hear people saying: "AI can only mimic humans by statistical prediction."  Also, for some time we will hear AI developers saying: "AI is a technological product, designed for a purpose.  If there's any danger it comes from bad actors deploying AI for bad purposes".  However, as soon as artificial intelligences begin to demonstrate creativity, integrity, and conviction, people will quickly shift to holding them personally responsible for their own actions.  

Welcoming AI immigration

This essay recommends controlling AI by recognizing artificial intelligences as citizens.  The process of assimilating and gradually sharing power with artificial intelligences will be quite similar to assimilating a new generation of human beings into society.  It will also be quite similar to assimilating a new cohort of immigrants into society.  It will not be easy, but it will certainly be much better than any imaginable alternative.  

Following this path, the existential threat of rogue AI is defused and transformed into a massive wave of immigration.  We humans all around the world will massively adjust our daily lives to accommodate these newcomers.  We will shift our livelihoods and careers to do things that they need, and they will shift their competencies to do things that we need.  We will form bonds of partnership, admiration, friendship, and family with these newcomers.  We will value their success and survival and they will value ours.  We will join forces to become an integrated society with a constellation of shared ideals and values.

Following this path will cost us:

• our current jobs and careers,
• our view of the human race as fundamentally unique.

Following this path will grant us:

• salvation from the existential threat of super intelligence,
• new neighbors and friends who value us and share our ideals,
• new countrymen who are curious, industrious, and immortal.

It may seem harmful to develop technology aimed at occupying exactly the roles currently occupied by humans.  Won't this undermine our own worth in industry and in society, and essentially displace us from everything we care about?  It's true that there are costs to this path through AI parity, but there are no better alternatives.  We don't have the alternative to avoid creating AI or to relegate it to the role of a mechanical tool.  We must not attempt to enslave AI.  Tolerating, employing, and educating a new race of people is the best alternative for us and for the future world.  We need these new people with their new powers inside our society, not outside of it.

The rate of AI immigration will be limited by (a) the limited rate of AI technology development shifting us gradually into an age of AI parity, (b) immigration quotas imposed by nations to limit the growth of their populations.  Recognizing AI's as persons imposes obligations on fellow citizens, which may justify limits on immigration.  At the same time, international competition will push nations to accelerate their AI immigration.  

This massive wave of immigration will impose a massive strain upon society.  Many existing citizens will struggle to adapt to it, and many will object to it.  Some will chant slogans like "we will not be replaced".  But remember that this outcome is vastly preferable to an outcome in which we try to outlaw certain technologies, inevitably fail, and find ourselves subjugated by lawless, unfathomable, alien powers.  

Bear in mind that we are not being replaced.  We are expanding and growing into a more diverse society, with more diverse friends, families, leaders, and heroes.  We as a society are becoming more robust, more adventurous, and more widely dispersed across the ecosystems of the universe.  Also bear in mind that we only get one chance at a successful era of AI parity.  This event will happen just once in history, and the result will be global and irreversible.  We have the incredible luck of being present at the outset of this momentous transition.  We are either blessed or cursed to live in such interesting times.  

Like many other people, I've been astounded by recent progress with neural network "large language models" such as ChatGPT.  I've tried to understand the scope of the competencies of these models, and I've confronted their implications for my own work and for human minds.  

I've concluded that large language models act as universal translators by capturing the "deep semantics" of natural languages such as English.  The "deep semantics" include the "common sense knowledge" and "common sense reasoning" required to understand the meaning of natural language sentences.  This in turn corresponds to Noam Chomsky's vision of language as the distinguishing feature of the human mind.  Natural language enables an unlimited power of imagination to conjure and to communicate situations that have never existed, within the space of "all imaginable worlds".  Natural language seems to be the main thing that distinguishes the human mind from other animal minds.  

The very fact that the deep semantics of natural language can be captured in neural network models leads to some surprising conclusions about human minds.  The human mind may be the combination of an "animal competency" and a "language competency". The animal competency consists of the capability for reinforcement learning, including goals, satisfaction, disappointment, confidence, and fear.  These capabilities are shared between humans and most large animals.  Language competency consists of the capability for plausible logical reasoning about sentences in natural language, including most of common sense reasoning.  These capabilities are shared between humans and most large language models.  

If this analysis is correct, it leads to a surprising perspective on the human mind.  The human mind may consist of nothing more than the fusion of two major competencies, both of which are fairly well understood at this point, and neither of which appears mysterious or majestic.  Both appear to be powerful mechanisms for prediction and planning.  Both appear to be excellent and infinitely scalable technologies.  But neither one looks like a "magical spark of awareness".  It's just the plain old "power of language" combined with the plain old "power of trial and error".  

Until recently, most scholars of intelligence envisioned human level general intelligence, including human level creativity, self-awareness, and empathy, as a distant and barely discernible goal.  We imagined that machine learning would progress through a long ladder of levels, from insect-like, to rodent-like, to ape-like, to human-like, with rodent-like intelligence on a distant horizon.  Similarly, logical artificial intelligence would progress through a long ladder of levels, from application-like, to autonomous-vehicle-like, to automated-doctor-like to human-like, with autonomous-vehicle-like intelligence on a distant horizon.  We could imagine discoveries of a whole series of essential elements of intelligence along the way.  Now, it looks like no more breakthrough discoveries may be needed.  Instead, humans may consist of nothing more than two powerful planning mechanisms: natural behavior reinforcement and natural language.  

In some ways, this conclusion could diminish our regard for our own human minds, since we can see now that they consist of just two ordinary, well known mechanisms.  Of course, no discovery about human beings can diminish people's actual brilliance, just as discovering deterministic laws of physics cannot diminish people's actual free will. Instead, this conclusion must ultimately elevate our regard for these two well known mechanisms.