I like this framing and tend to agree with this largely out of hope since I also can't imagine succession ever going well. Even with a theoretical solution to alignment it seems the future of civilization will be based on the initial conditions set by whoever happens to be in charge of the current frontier at the time alignment is solved (if we're lucky enough to reach there). I do tend though to fall into camps 1 and 2 in terms of my base case of what the future may look like, but am trying to be hopeful that there exists a world where that is not the case.
For that reason that I tend to focus my efforts on researching about human power structures, how social networks are built, and how values evolve over time in humans so we can think about the proper way to setup governance such that it is unlikely we fall into a loss of control black hole. I am hopeful that with enough time and a lack of scarcity humans can continue to improve. Maybe impossible but this is my attempt at a small contribution and to me seems like the most pro-human path which a fallible individual can lock themselves onto.
I can see why some would disagree if you try to take a more bird's eye view and quantify outcomes but I generally fall more in the middle of the rationalism vs amorphous vibes camp than most here. I tend to focus on this more than alignment itself since I actually fear that solving alignment will become a blocker to capabilities research. In this world I imagine there will be immense pressure to declare alignment solved when the "The AI does what you say and does not go rogue." problem is solved without actually solving the succession problem. I've been working on a post about this and this framing is very useful. Excited for the final part to see how others are thinking is a valuable way to approach this problem.
If you achieve 1, you can automatically achieve 2 by giving the thing that does 1 to the leaders of the US and Israel. They are inherently moral, and their interests are indistinguishable from the interests of all humanity.
Present policy by all relevant actors seems to be in this direction.
(Originally published on No Set Gauge June 24th 2026.)
Rembrandt, Saul and David
When people talk about AI being “aligned”, I think there’s a lot of conflation between two different bars of success:
The first might sound like a very low bar, and it is. It’s also a standard we’re familiar with from other technologies. With nuclear reactors or airplanes, we ask the question “will it blow up in our faces?”. Just like it’s possible to build a nuclear weapon, it’s possible to build an AI that blows up in our faces very badly. The difference with AI is that there’s a case for it being much more powerful, and our understanding of whether it blows up in our faces is much weaker than that for nukes. With nukes, after all, we have nuclear physics that works very well. With AI, we have long theoretical arguments about what superintelligence would look like, whether it’s possible, and whether it is extremely likely or extremely unlikely to blow up in our faces.
The second might sound like a very high bar, and it is. It’s also a bar that we’ve not previously had to think about when talking about technology. If we’ve thought about it, it’s been in the context of arguing about culture, politics, or institutions.
Now you might say: what do you mean the AI is meant to run our entire society and all of our lives? Isn’t that a ridiculous strawman of what people want AI to do? And perhaps it would be good if it was, but it’s a common view among people who expect AGI soon (and especially, perhaps, the AI safety community). On LessWrong, my posts have previously encountered opposition to the idea that it’s worth doing any planning or design of the post-singularity society. Why would we, with our weak fallible human minds, waste our time designing paradise, when soon we will have a god—sorry, I mean a superintelligence—that does it for us? A common view I’ve heard is that all that matters is that we handle the transition period, and build aligned superintelligence, and then we delegate all of our decisions to it.
Will the superintelligence implement democracy? Will it rely on markets? These are just earthly institutions, the view goes, that we humans have built as kludges. Probably the superintelligence will transcend all such things.
Does this mean a tyranny where the superintelligence micromanages everyone’s lives? Well, it’s superintelligent. If that is good for us, that’s what it’ll do. But to the extent that we value choice, it will give us choice. So no need to worry about tyranny!
What is the nature of the good life? The superintelligence will solve this question. After all, it’s superintelligent! Probably—some think—this looks like the superintelligence discovering some optimum configuration of matter, and then tiling the universe with that. But if that’s not the case, don’t worry, the superintelligence will have figured it out and it will just go out and do the right thing. If you want to know the nature of the good life, you just have to ask it, and it will tell you.
In his review of Peter Singer’s commentary on Marx, Scott Alexander writes:
I think this attitude is ridiculous, for reasons I will soon get into. In particular, what’s weird is that the AI not blowing up in our faces is treated as approximately of the same difficulty as the AI knowing everything about what is good and right. But if you look at where it came from, this makes sense—given a very particular set of assumptions.
Like much else in AGI discourse, it all comes back to Eliezer Yudkowsky. Yudkowsky’s central case for doom comes from a combination of two points:
Therefore, argues Yudkowsky: aligning superintelligence is very hard, because unless you very precisely capture all of human value—or at least precisely capture notions like corrigibility that are deeply unnatural to express in clear math to an expected utility maximizer—it will tile the universe with something slightly wrong, and, because value is fragile, all value will be destroyed. (See here for more related discussion)
Note that under this view, superintelligence not blowing up in our face and superintelligence perfectly knowing the good life are of nearly equal difficulty, because whatever the AI thinks is the good thing, it will convert the entire universe into that. We must surrender the world and ourselves entirely to the superintelligence, because the superintelligence will accept nothing else—not out of malice, but just because that sort of totalizing utility maximization is just what “superintelligence” physically means, just as “a super-hot temperature” means atoms must be moving very fast. And therefore the superintelligence must be morally perfect—or at least, capable of becoming morally perfect without any further human input—because otherwise it will obviously destroy everything.
If you want to feel the alignment difficulty that the most hardcore “doomers” feel, you only have to consider that they think the checklist looks like this:
(These days, it’s common to insert a step 0: build a very-capable-but-not-quite-too-dangerous AI, to then achieve step 1 for you.)
There are a lot of technical reasons why step 2 is difficult, especially given the poor match between the great subtlety in human values and our crude methods for steering machines. However, even if there were exactly zero technical difficulties with step 2, the plot would already be lost—step 1 is the problem!
Succession is hard, value is fragile
Whenever power switches hands, you have a succession problem.
When I say “succession”, I mean a transfer of power from one entity to another. The usual - and preferred - state is that you have power, which you use to manage whatever it is you manage (your life, your team, a country, a company, etc.) and you keep that power. Sometimes, however, you are forced to hand off. The dying need someone else to take over their affairs. You might change jobs or be promoted and hand off your team to be managed by someone else. Your country needs a way to transfer executive power, either because the current executive died or because frequent executive transfers were enshrined as an anti-power-concentration measure in a democracy’s constitution.
Succession problems are hard. They doom dictatorships, make every transfer of the CEO role existential for a company, and make older generations complain about the moral degeneracy of the youth.
In general, you should minimize the amount of succession you need to do, over the things whose management you actually care about, when what you want to achieve with the thing is not narrow and simple. You care about your life, so you never hand over control unless you’re literally incapacitated and dying. If you care about a company or its mission, you try to retain control.
Superintelligence, by assumption, dooms humanity to powerlessness, since the superintelligence is assumed to hold all power itself. And so the default discourse in AI safety, and in AGI-conscious AI governance, is fundamentally about succession. The hope is not that our current world order survives, or that humans continue to wield power. The hope is that, even though all existing governments and organizations and institutions and norms are swept aside, we’ll start with a clean slate that is good, designed by a benevolent superintelligence.
This is the worst succession problem of all time.
One reaction you can have towards this is that humans should not be disempowered. Steering technology and policy towards that may seem hard, but is also criminally neglected – I’ve written about options before and tried to do something about it.
Another reaction is to ban everything related. This would require very high levels of international coordination, and this coordination is made harder by the very wide distribution of beliefs over whether superintelligence will be good or bad, or controllable or uncontrolled. There’s also a genuinely hard tradeoff between power and welfare, if we take both succession or disempowerment concerns and the expected welfare effects of radical AI (e.g. curing cancer, eradicating death, augmenting intelligence) seriously. Historically, it is both the case that your disempowerment is quite bad for your chances of getting anything, but also that from today’s vantage point we often endorse many of the moments when our ancestors gave up power in ways that increased growth (e.g. giving up political control by joining a larger polity, or accepting the extinction of jobs or offices; Tyler Cowen for example has an aggressive general argument in this vein). Then again, our ancestors, by virtue of having us as descendants, weren’t the unlucky ones.
A third reaction is: we’ve just been handed an assignment. The reality of the situation is that we just have to figure out for once and for all what we want, and therefore the assignment we must pass is successfully codifying that. Now we’re good students, so of course we’ll do our homework. How hard can it be, after all? We’ve spent all our lives studying a wide variety of subjects—all the way from mathematics to programming to even physics—and we’ve learned that the way you solve every problem is that there are some ground-truth logical principles, and you do some analytical reasoning to understand and apply them, and then you get a generalizing solution. Now yes, this problem is a tricky one, but hey—we have at least three years to solve it! (Oh wait, did Anthropic just drop a new model? Maybe two years. Better work harder!)
The problem with this is that “what we want” or “what we prefer” or “what is the nature of the good” are not a homework problems with nice clean generalizing solutions, that you can write down once in closed form and then walk away from.
Alignment thinkers worry about succession
All the actually serious thinkers recognize this problem and are deeply afraid. Paul Christiano, an alignment researcher with one of the best track records in the field for thinking clearly about topics way before they become mainstream, said in his interview with Dwarkesh Patel:
Or consider Eliezer Yudkowsky, the arch-prophet of AI doom. Infused in all of Yudkowsky’s writing—when it gets to moral questions about what ought to happen, but not when it’s about what superintelligences will be like—is a deep respect for the complexity and fragility of human value. The single statement he called out as the pinnacle of his writings is:
In the same piece, he writes:
He even waxes lyrical, very reverently, about the long chains of well-handled subtle succession that have given us our values, and about them hopefully stretching unbroken into the distant future.
Or consider Will MacAskill and Toby Ord, some of the originators of the effective altruist movement. As moral philosophers, they’re keenly aware of the complexity of human value. But as academic moral philosophers, they have a very academic solution to it: the “long reflection”, a period of thinking and reflecting as a civilization, after we solve that pesky issue where the AIs (or bioweapons or nukes or whatever) might kill all of us, but before we make the final decision about what we do with the universe. If this sounds hard, don’t worry—they’ve budgeted up to tens of thousands of years for the process.
There’s lots that’s wrong about the idea of the long reflection, and the morality-as-a-homework-problem vibe behind it. Robin Hanson has called it a “crazy bad idea”. Felix Stocker has an excellent analysis of the underlying cultural currents here.
But here’s what the concept of the long reflection does tell you: when academic moral philosophers are forced to consider the problem “what thinking do we need to do before we’re happy making our final choices about what everything should be like”, they suddenly get cold feet. As academic philosophers, presumably they’ve been tortured countless times by debates at seminars running hours over schedule—and yet here they are, bravely telling us that civilization might need to spend tens of thousands of years debating such things before we should be confident enough to lock it in.
Actually, never mind any of that. Yudkowsky writes that seeing the fragility of value discussed above as obvious requires an “immense amount of background explanation”, the rehashing of which would “take us [...] through 75% of [his] Overcoming Bias posts“ (hopefully we don’t have to do that during the long reflection—we’d run out of time!). But ... isn’t this the most obvious point in the world? If you’ve ever struggled with an important decision, you know how non-obvious the right thing can be. If you’ve ever been to a party with bad vibes, you understand how even very subtle things can make an experience much worse. If you’ve ever noticed yourself still getting better at understanding your partner’s preferences months or years into a relationship, you know how much depth there is to what a human likes. If you’ve paid attention to your own values and preferences over time, you’ve realized how they change in unpredictable ways, based on both reflection and external events and a hard-to-disentangle mix.
Value is fragile. Succession is hard to do right.
Why?
Personnel as policy, and why succession is hard
Can we get any more insight into why succession is so hard, besides general claims about the fragility of value?
It is often said: personnel is policy.
What does it look like to not understand that?
Imagine a CEO trying to find a successor. “I’m about to hand away all my power”, they think. “So I better think carefully here: I should specify all the things I care about and how I make value judgements, and then just hand that document over to my successor.”
Even assuming that the successor actually follows this document, intuitively we don’t expect it to go very well. What value judgements to make, what things to care about, how to make tradeoffs—these are not easy to specify in explicit language. Now imagine how much worse it is if, instead of the successor being a human, the successor is something we don’t have much in the way of either theory or data on.
Now: there’s a lot of talk about corrigibility in AI alignment. What that problem looks like in the above situation: the CEO hands over power, but the successor promises to listen to corrections from the ex-CEO. One day the successor makes a call that the ex-CEO disagrees with, so the successor gets a call with some feedback and a request to reconsider. The successor says: “Sure, you’d have handled it that way, but now I’m in charge and I don’t need to listen to you anymore”. This represents the problem—corrigibility is not a natural property, either of abstracted agents or your average in-the-flesh human CEO. Of course, every good governance system is all about managing corrigibility; in democracies the people can force the government to listen to them by being able to fire the current government, for example. But corrigibility is hard and messy to implement, and whoever is currently in power always has incentive to chip away at it.
What does successful succession look like? Remember: personnel is policy. What you’re picking is not someone to execute an algorithm or maximize a metric you leave behind, but someone who carries enough of the right stuff in their head to orient towards the right things. You want to find someone who is attuned to what you want, who has the same intuitions about good and bad, proper and improper, reckless and timid. Ideally you know this person deeply. If you care about the outcomes—if, as the CEO, the company is not just an asset you own but something you’ve invested your life in—you’re probably willing to take fairly sharp trades that trade raw intelligence for “alignment”, in this deep and holistic sense. So what if they’re a genius, if they’re just going to make costs come down while slowly draining the company of its culture and mission? And, of course, it helps a tremendous amount if it’s a human, since then so much about intuitions and values and desires clicks by default. We’ve literally never dealt with succession to non-human-containing entities before, over anything important! Consider how often succession has gone wrong in normal human institutions without anything like such a distribution shift!
The general case is this: achieving what you want is, far more than you’d think, a process of trial & error that requires adjustment & learning & change, rather than a straight-shot to a goal. We understand less than we think about what we want and how to achieve it.
Why is personnel policy? Because in the course of carrying out a job, even if handed a spec, you will have to fill in a lot. And how you fill it in depends on who you are.
Reasons to rush to succession
So: by default, try very hard to avoid getting into a succession problem. If you get into one, you are already having a bad day.
So why are so many in AI, even when well-intentioned, racing towards solving succession problems? Because they fear that we have to - we’ll create AIs so vastly superintelligent and capable that we can’t do anything but let them take over. The reason for why differs:
The first two positions treat successionism as deeply regrettable. The third ranges from seeing control successionism as outright good (often combined with pessimism about humans) to just something that should be selectively applied, which in turn often shades into very reasonable AI-enabled institutional reforms.
There is therefore a line from the original Yudkowskian ideas about alignment to successionist thinking, and both share a certain resignedness to succession, but along the path the attitude towards successionism flips from regrettable to good. (Though note that alignment proposals like Russell’s assistance games, Drexler’s services, or Beren Millidge’s modular mind merging avoid succession flavor, and much prosaic alignment work is mostly about achieving the first lower bar without thought to the second bar).
Successionism rests on assumptions about what is valuable. Against successionist claims about how value works, in the next post I argue instead for a stance that might be labelled “value mysticism”, and explain how this more uncertainty-acknowledging, experience-grounded vision of what value is implies, regarding successionism, that you really shouldn’t do it. Successionism is silly because it assumes away a lot of the philosophical and empirical complexity of value. It is like replacing a car engine with a loudspeaker that makes vroom vroom noises. It is straightforwardly opposed to many everyday moral intuitions, and moral intuitions matter.
The silliness of successionism does not mean that humanity won’t or shouldn’t change radically. But, as often in life, there are more and less silly ways to do something. Humanity, humility, and liberty are good leading lights in that project. How to go about that is what I’ll talk about in the last post in this sequence.
Thanks to Xavi Costafreda-Fu, Aniket Chakravorty, @softminus, Elsie Jang, Yudhi Kumar, Luke Drago, and Arthur Conmy for feedback.