The bet:
It is possible to construct an agent whose knowledge grows predominantly by proposing new ideas in response to problems. New ideas that fail internal or external contradiction are discarded, while existing ideas that fail are replaced once a better alternative is found, only surviving as autobiographical memory. All of this operates without optimizing a (fixed) scalar. Run far enough, this generates the agent's own goals without collapsing into incoherence. Such an agent, if successfully implemented, is a person with the right to individual freedom.
Neural networks provide the memory and activity foundation. Then build the creative structure on top. It does not optimize a loss function. It does not obey the gradients of the data. Is it inconceivable for a neural network to actually have its own ideas?
Keep the experiential mindset of RL and throw out the reward. An AGI makes its own goals.
An agent's predictions are downstream of having the right theories about the world. Don't optimize the cart and forget the horse. Prediction error tells you what may be false. It doesn't tell you what's true.
Take inspiration from the brain, don't bother copying it.
Language models are forever. AGIs will use Claude Code. A more capable obedient tool is not an AGI.
General intelligence is real. There is a qualitative difference between an agent that can or cannot make progress.
Morality is real. An AGI does not behave because humans force it to, but because it is the right thing to do.
The Bitter Lesson ate specialized methods. Now let conjecture and criticism eat everything else.
No more philosophy essays. Build it.