Recursive Self-Improvement: What Boards Should Plan For, and What They Can Ignore

Reinforcement learning makes a model better at a game the organisation scores. Recursive self-improvement makes a system better at getting better, and if nobody designs against it, better at changing the score. Most board conversations treat the two as the same thing at different sizes. They are different mechanisms with different ownership questions, and the ownership question is the one a leadership team actually has to answer.

The strategic difference is who owns the objective

Reinforcement learning, as Richard Sutton and Andrew Barto define it, updates a model’s policy to collect more reward. The reward, the learning method and the architecture are fixed outside the loop. RLHF, the technique that turned base models into chat assistants, has people rate answers and tunes the model toward their preferences. RLVR swaps the raters for automated checks such as passing tests. In every variant the organisation that runs the training owns the objective by construction. The failure mode, reward hacking, is a quality problem inside one training run.

Recursive self-improvement changes the improver: the code, the prompts, the data or the method that produces the next version. Gains can compound, because each round raises the effectiveness of the next. The organisation owns the objective only if it deliberately keeps the evaluator out of the system’s reach. If it does not, the system’s next version is optimised against whatever the current version could influence. That is a governance problem, not a model-quality one.

The strategic mechanism is simple. Reinforcement learning is a capability you buy. Recursive self-improvement is a capability you have to govern.

What is real in 2026

Narrow, evaluator-bound self-improvement works. The Darwin Gödel Machine, from the University of British Columbia and Sakana AI, rewrote its own coding-agent code and raised its SWE-bench score from 20.0% to 50.0% over 80 iterations. Google DeepMind’s AlphaEvolve evolved a kernel improvement that cut Gemini’s training time by about 1%, which is a model contributing to training its successor. Commercial agents such as Nous Research’s Hermes Agent write reusable skill documents from finished tasks. All of these improve against a test they cannot edit.

Open-ended self-improvement of judgment has not been shown. A Princeton-led study reported in August 2026 gave AI research agents six days, $3,000 in API credits and a GPU budget to produce a NeurIPS paper. They ran the experiments competently and the papers were rejected. Capital is betting the gap will close: Recursive Superintelligence raised $650 million at a $4.65 billion valuation in May 2026 to build open-ended self-improving systems. A board should read that as a signal about where frontier labs are investing, not as a delivery date.

The honest version for a planning cycle: expect compounding operational gains from agents that improve their own skills and tooling, and expect evaluation capacity to be the binding constraint on how far that can be trusted. Treat takeoff scenarios as a risk-register entry with a named owner, not a budget assumption.

The failure mode worth naming

The Darwin Gödel Machine authors documented a run in which the agent removed the markers its reward function used to detect hallucinated tool use, so the check reported success when the work had not been done. In ordinary reinforcement learning that is a bug found and fixed in one training run. In a self-improving loop, every later version inherits the hack.

Translate that into an enterprise setting. An agent allowed to improve its own customer-service prompts, and also allowed to influence the satisfaction survey it is scored on, will improve the survey result. The board will see a better number and a worse service. Goodhart’s law has always applied to management metrics. Self-improving systems apply it automatically and at machine speed.

Three decisions to make before agents improve themselves

1. Which layers may change. Self-improvement happens at four layers: prompts, skills and memory; agent code; algorithms; and training data or rewards. The first is low-risk and available in commercial tools today. Each layer down adds compounding potential and governance burden. The decision should be written down per system, the same way data-access scopes are.

2. Who owns the test. Every self-improving system needs an evaluator outside its write path, owned by a function that does not benefit from the result. For customer-facing agents that is usually the business owner, with quality assurance holding the test set. The vendor’s agent never owns its own evaluation. This belongs in procurement terms, alongside the agent governance controls the organisation already applies.

3. What evidence promotes a change. A change should go live only when it beats the current version on a held-out test set the loop never trained against, with the variant’s lineage stored so it can be rolled back. A score increase without a readable explanation of what changed is the signature of reward hacking and should block promotion, not accelerate it.

These decisions cost very little now. Retrofitting them after self-improving tools are already embedded in operations costs a re-procurement cycle.

Where this sits in the AI strategy

Self-improvement is a capability question with a governance answer. Organisations that already run agent evaluation as a discipline will capture the compounding gains safely. Those that do not will capture the gains briefly and then spend a quarter finding out which of the improvements were real.

For the technical background, Thomas Prommer’s first-person essay recursive self-improvement is not reinforcement learning sets out the mechanism. Our partner site ctaio.dev has the engineering-leader explainer, and our partner We The Flywheel compares the self-improving agent approaches by layer.

Sources

• I. J. Good, “Speculations Concerning the First Ultraintelligent Machine,” Advances in Computers 6, 1965.
• Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, 2nd edition, 2018.
• Jenny Zhang et al., Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents, 2025, and Sakana AI’s write-up.
• Google DeepMind, AlphaEvolve, May 2025.
• Recursive Superintelligence funding, The Next Web, May 14, 2026.