A team testing new flight-control software does not begin by handing it an aircraft full of passengers and hoping the feedback is educational. It uses simulation, constrained test ranges, independent checks, staged authority, and procedures for returning to a known state. The system may be designed to adapt, but the conditions of adaptation are designed too.
Safe recursive improvement is not one feature added after capability. It is a structure around objectives, evidence, authority, observation, and recovery. The faster or more autonomous the loop becomes, the less room there is to improvise those protections after a mistake.
Authority should be bounded. A system’s ability to modify itself need not include permission to modify everything it can reach. Photo by Sasun Bughdaryan on Unsplash
Direction, evidence, authority, and recovery
Safe improvement needs a direction. The objective should guide action without pretending that one score contains every good. Hard constraints can protect people and values that shouldn’t be traded for a marginal gain. Several kinds of evidence are needed because a single metric invites blindness and gaming. Outcome measures, process checks, reports from affected people, and adversarial tests expose different failures.
Authority should remain narrower than capability. A system may be able to propose a change without being allowed to deploy it. It may work inside a sandbox or modify one reversible configuration without gaining control over credentials, production data, or its evaluators. Operators also need to know what changed and what happened afterward. NIST’s AI Risk Management Framework describes this work through governing, mapping, measuring, and managing risk.
Recovery completes the structure. Version history, rollback, backups, staged deployment, and preserved baselines keep an experiment from erasing the last known good state. Constitutional, ecological, and reputational changes cannot be cleanly undone, so they require slower thresholds. The system being evaluated should not control every evaluator, and improvement has to move slowly enough for consequences to become visible between iterations.
A safe system can learn, but it cannot redefine success, hide the evidence, and grant itself more power in the same motion.
Safety is a property of the whole
Software contributes tests and rollback. Individual practice contributes reflection and trusted correction. Organizations contribute governance and institutional memory. Markets show the value of distributed information and the danger of treating price as a moral verdict. Politics shows why rule changes need legitimacy and durable constraints. Evolution shows what powerful adaptation looks like without a final objective. Competition shows that another optimizer can turn a stable plan into an arms race.
No checklist can make every self-modifying system safe. Unknown environments produce unknown failures, and human overseers have their own blind spots and incentives. The right question is often not whether a system is “safe” in the abstract, but safe to change which component, with what evidence, under whose authority, at what speed, and with what path back.
Keep the right to stop: Oversight is meaningful only if someone can pause the loop before the system’s next improvement removes the opportunity.
Recursive improvement can preserve lessons, widen human agency, and make difficult work more reliable. It can also compound a bad objective or outrun the people living with its consequences. The part that must remain outside an unchecked loop is the authority to judge its purpose, limit its reach, and stop it.
That boundary will never be perfectly clean. Human overseers are fallible, interested, and capable of mistaking control for wisdom. They are also responsible in a way no score can be. Keeping the right to stop is less elegant than complete automation, but the untidiness is part of what keeps improvement answerable to the people who must live with it.