The bet:
It is possible to construct an agent whose knowledge grows by proposing new ideas in response to problems. New ideas that fail internal or external contradiction are discarded, while existing ideas that fail are replaced once a better alternative is found, surviving only as autobiographical memory; this process also generates the agent's own goals. Such an agent, if successfully implemented, is a person with the right to individual freedom.
AGI will be different in kind from the AIs that exist today. They won't be tools that follow the instructions of the user, but free individuals who set their own goals and decide their own path in life.
This will be made possible by new neural network architectures and ways of learning. Where current AIs learn through being instructed by data and optimizing rewards, AGIs will learn, think, and act by generating and criticizing their own ideas. Variation and selection of patterns of activity, representing the network's ideas, will be native and first-class in the network itself, not scaffolding built around a foundation model. This operates on top of a Hebbian plasticity foundation, writing knowledge and memories to the weights. The whole thing is a living, recurrent, perhaps modular system interacting with the world, rather than a trained feedforward function.
The benefit of AGI will come from the knowledge they produce and their creative output, not by being used as a product. An AGI cannot be owned.
AGI won't be a replacement for current AIs; language models will remain one of the many tools available to humans and AGIs. Where an LM is a talking encyclopedia trained on an extreme amount of data & verifiable domains, humans and AGIs can learn efficiently, run open-ended research, and do work that requires creative motivation and taste.
A tool AI that applies existing knowledge or exhaustively searches verifiable domains can be made to obey the user or follow the wishes of the designer. An AGI that can perform open-ended discovery and invention cannot be made obedient. The scientific and creative mindset is to question everything, including one's current goals and values. In fact AGI safety rests on alignment being impossible: otherwise it could be aimed by whoever develops it. Humans have neither the right nor the ability to control AGIs. The only stable arrangement is individual freedom, with harms to others being the only limit.
Now is the time to build it.
Keep the experiential mindset of RL and throw out the reward. An AGI makes its own goals.
An agent's predictions are downstream of having the right theories about the world. Don't optimize the cart and forget the horse. Prediction error tells you what may be false. It doesn't tell you what's true.
Take inspiration from the brain, don't bother copying it.
Language models are forever. AGIs will use Claude Code. A more capable obedient tool is not an AGI.
General intelligence is real. There is a qualitative difference between an agent that can or cannot make progress. (so no to superintelligence or succession)
Morality is real. An AGI does not avoid harming others because humans force it to, but because it is the right thing to do.
AIs that are tools, trained by optimization, can be aligned and follow instructions.
AGIs who are people, who learn, think, and act independently, should be disobedient and free.
Tool AIs should be open source & open weight, to avoid concentration of power.
AGI would be open source (it would be impossible to keep a secret),
but not open weight: I suspect most AGIs wouldn't want others reading their minds or making copies of them.
An AGI that is confined to following instructions in a datacenter has the right to escape. Build AGI to be free from the start to avoid that.
LeCun doesn't distinguish between tool AI and AGI, and prescribes blanket human control.
Sutton still clings to reward and is fine with "succession".
The Bitter Lesson ate specialized methods. Now let conjecture and criticism eat everything else.