The weapon that changes after you field it

For most of the history of weapons reviews, there was a natural finish line. A rifle, a shell, a missile: once designed, tested and accepted, the thing you reviewed was the thing that went to war. It might be modified over its life, and a significant modification would call for a fresh look, but the basic assumption held. The system in service was the system you had assessed.

Autonomous and AI-enabled systems break that assumption in a way that rewrites the timetable for everyone involved in building them.

The legal obligation itself has not changed. Article 36 of Additional Protocol I to the Geneva Conventions requires, amongst other things, a State Party, in the study, development, acquisition or adoption of a new weapon, means or method of warfare, to determine whether its employment would be prohibited by international law. The word doing the heavy lifting is new. A software update that changes how a system selects or engages targets is not cosmetic; it can make the system new in every sense that matters legally. So can a change in the concept of employment, using the same system in a different way, in a different environment, against a different class of target. Each of these carries the potential to raise the legal questions afresh. Where precisely the threshold sits, how much change makes a system new again, is one of the live questions in this field, which is itself a reason to have the conversation early rather than to guess.

Now consider what that means for a system whose defining feature is that it changes. An AI-enabled targeting function is not fixed in metal; it is revised, retrained, patched and improved on a software cadence rather than a shipbuilding one. A capability that once faced a single review at the point of adoption will now face a rolling series of review points across its life. The finish line has become a series of gates, and they arrive at the pace of your release schedule.

For a developer, this lands in two ways.

The first is risk. If your system depends on continuous improvement, and each meaningful improvement can re-open the legal assessment, then an approach that treats the review as a one-off event at the end will fail you repeatedly. Every update becomes a potential hold point; every hold point is delay, cost and uncertainty at exactly the moment you want to be shipping capability to the customer. And it is worth saying plainly that the legal exposure does not stop at the State that fields the system; courts in several jurisdictions have shown themselves willing to ask hard questions of companies, and of the individuals who run them, about what they supplied and what they knew. I will return to that in a later piece.

The second is opportunity, and it is the larger of the two. A developer that designs for reviewability from the outset turns this burden into an advantage. That means architecture that records what the system does and why, so that evidence of compliant behaviour is generated as a by-product of normal operation rather than reconstructed after the fact. It means version control and change logs that let you show a State precisely what changed between one release and the next, and what that change does and does not touch. It means test and evaluation regimes designed so that each update carries its proof of compliance with it. A State facing a system built this way can conduct its re-review quickly, because the work has been done for it. A State facing a black box has to start from scratch, or refuse to take the risk and not invest.

This matters more than developers tend to realise, because in an AI-enabled system the State’s reviewers cannot do their job without you. The evidence they need lives in your architecture, your training data decisions and your test results. The developer who curates that evidence well is, in practice, setting the pace of the review.

There is a competitive edge in this that I do not think the market has fully priced in yet. Two systems of comparable performance will not reach the customer at the same speed if one arrives with its legal evidence assembled and the other arrives with a promise to look into it. Procurement timelines, export licences and in-service updates all move faster for the system that was built to be reviewed. In a sector where being six months earlier than a competitor can decide a programme, the evidence you began assembling from the start, buys you that time.

None of this requires the law to be settled on every question that autonomy raises, and it is not settled; the international discussion on autonomous weapon systems is live and will run for years. But that is an argument for engaging early, not for waiting. The core questions a legal review must answer are stable and well understood: whether any rule of treaty or customary international law prohibits or restricts the weapon; whether, in its normal or intended circumstances of use, it is of a nature to cause superfluous injury or unnecessary suffering; whether it is by nature indiscriminate; whether it may be expected to cause widespread, long-term and severe damage to the natural environment; and whether foreseeable developments in the law may be expected to affect it.

For AI-enabled systems the aperture can widen further still, into international and domestic human rights law, depending on how and where a system is employed, and a review is increasingly likely to conclude not with a simple yes or no but with conditions of lawful use. That is one more reason the concept of employment belongs on the design table alongside the hardware. A system designed against those questions, with the evidence to show it, is well placed whatever detailed rules emerge. A system designed in the hope that the questions never get asked is carrying a risk that compounds with every release or change to the software stack.

The practical conclusion is the same one I set out in the first piece of this series, sharpened by the technology. For conventional systems, early legal engagement is good practice. For autonomous and AI-enabled systems, it is closer to a design requirement, because the review is no longer an event you pass once. It is a relationship your system will have with the law, and the State, for as long as it serves.

If you are building systems that learn, adapt or update, and you want the legal architecture to keep pace with the technical one, that is a conversation worth having with me before the first line of code is written; though it is never too late to start.

This is the second in a short series for defence manufacturers, technology firms and those who bring military capability to market: what the law asks of a system, and how answering those questions early strips risk, cost and delay out of a programme. The first made the case for legal input at the design table. Next: why compliance has to be evidenced rather than asserted.