Every program I run has a change-control board, and every one of those boards would reject the following request: swap the engine that produces our status summaries, risk scores and draft plans for a different one, effective immediately, no notice, and nobody will tell you what's different. Yet that is exactly what happens when the vendor behind your team's AI assistant ships an update. The model gets retrained or retired, the behavior shifts, and on Monday morning the same prompt returns a different answer. Nobody raised a change request. Nobody signed anything.
I'm not complaining about vendors. Models improve, and the new one is often better. My point is narrower: an AI model that contributes to your program record is a component of the program, and components that change without a record are the oldest source of "it worked last month" in delivery. We already know how to govern this. We just haven't noticed that the thing needing governance has moved from the software we deploy to the tool we use to run the program.
The baseline you can't see move
Think about what your team has quietly built on top of the assistant. A prompt that turns a ticket export into a weekly summary. A template that scores risks from one to five. A routine that flags schedule slips above some tolerance. Each was tuned by someone, tested against a handful of examples, and then trusted. That tuning was done against one specific model, on one specific date. Nobody wrote down which.
When the model changes, three things go wrong, and all three are quiet.
1. The tuning is lost without a trace
The risk-scoring prompt that produced sensible numbers now produces slightly different ones. Not broken, not obviously wrong, just shifted. A 3 is now a 4 on some items and a 2 on others. The people who tuned it have moved on to the next thing, and nobody is re-checking, because nothing announced that a recheck was due.
2. Your trends become artifacts
This is the one that worries me most. Programs lean on trend lines: risk exposure over time, slip rate over time, sentiment over time. If the instrument that generates those numbers changes partway through the series, you now have a step in the chart that reflects the tool and not the program. A steering committee will read it as the program getting better or worse, when the real explanation is sitting in a vendor release note nobody on the program has read.
3. Nobody can reproduce the decision
Six months on, someone asks why the team rated a vendor dependency low in March. You still have the output. You don't have the model that produced it, and running the same prompt today will not give you the same answer. The record says what the machine concluded and is silent on the machine. For anything that touches a contract, an audit, or a post-incident review, that's a gap, and it will be found at the worst moment.
A tool that helps write your record is part of your configuration. If you wouldn't let a vendor swap your scheduling engine without notice, don't let one swap the thing that writes your risk scores.
Sign for the equipment
In the Army you don't carry equipment you haven't signed for. A hand receipt says what you hold, what condition it's in, and who is responsible when it goes missing. It sounds like paperwork until the day an inventory comes up short and the only thing standing between you and a very bad conversation is a list. The same discipline works on AI tooling, and it takes about an afternoon to set up.
Start with an inventory. List every place on the program where an AI model produces something that enters the record or informs a decision: summaries, scores, drafts, classifications, flags. For each one, write down the tool, the model or version if you can see it, the prompt or configuration in use, and the name of the person who owns the output. Most programs find they have more of these than anyone realized, and a few that nobody owns at all.
Next, pin what you can and watch what you can't. Some tools let you fix a model version; use that where it matters and accept the maintenance cost of upgrading on your schedule rather than the vendor's. Where you can't pin, subscribe to the release notes and assign someone to read them. It's a dull job and it's the whole control.
Then keep a small set of regression cases. Ten or fifteen real inputs from your own program with the answers your people agreed were right. When the model or the prompt changes, run them and compare. You aren't looking for perfection. You're looking for drift, and for the cases where the new answer is better and the old trend line needs a footnote saying where the instrument changed.
Finally, put it through the process you already have. A change to a model that scores risks or drafts the board report is a change to the program's reporting method. Log it, have the owner approve it, and annotate the series. The change board doesn't need a new form. It needs to be told this category of change exists.
- Inventory every place an AI model feeds the record or a decision, with the tool, version, configuration and a named owner for each.
- Pin the model version where the tool allows it, and upgrade on your schedule. Where it doesn't, assign someone to read the release notes.
- Keep ten to fifteen real inputs with agreed-correct answers, and rerun them whenever the model or the prompt changes.
- Annotate trend lines when the instrument changes. A step in the chart that comes from the tool should never be read as news about the program.
- Route tool changes through the change control you already run. The owner approves the swap the same way they would approve a new scheduling engine.
The bottom line
The problem with a silent change isn't that it's large. It's that it's silent, and a program run on measurements can't afford an instrument that moves without telling anyone. Models will keep changing, and mostly for the better. That's fine, as long as someone on your program can say which one produced a given number, when it changed, and who agreed to the change. Know what you're holding, know who signed for it, and the next update becomes a line in the log instead of a mystery in the graph.