Notes

Maintaining AI After Launch, What Go-Live Actually Starts

What does maintaining AI systems involve after go-live?

Maintaining AI systems comes down to five habits: watching a small set of numbers, reviewing a sample of decisions, controlling every change to prompts and models, working through the exceptions, and keeping one named owner responsible for all of it. None of it is heavy. All of it is easy to forget once the launch is over and the excitement has moved elsewhere.

The reason it matters is that AI work does not wear out the way a part does. It keeps running. It just gradually becomes a little less right, and nobody notices until a customer does.

Why does an AI system need ongoing attention at all?

Because the world it works in keeps changing, even when the system does not.

Traditional software does exactly what it was built to do until someone changes it. A rule-based workflow, the kind we compare in AI agents vs Zapier, tends to break loudly with an error. An AI agent reads language and makes calls, so it is sensitive to things outside its own code: the questions customers ask, the products you sell, the format suppliers use, and the behaviour of the underlying model, which the provider updates on their own schedule.

So the work after launch is less like servicing a ute and more like supervising a good new hire. They are capable, they are mostly right, and they still need someone checking in.

What should you monitor once it is live?

Monitor the same two or three numbers you agreed before the build, plus a short weekly look at volume, error rate and how often the system handed work to a person.

The original success measures tell you whether it is still doing its job. The operational ones tell you early when something is shifting. A sudden rise in hand-offs to people often means a new kind of input has appeared. A sudden fall can mean the system has become overconfident, which is worse. Set simple thresholds on each, with an alert to the owner when one is crossed, and keep the logs long enough to look back at what happened.

Against all of it, keep the original baseline. Without a baseline, every review turns into an opinion.

What is drift, and how do you catch it?

Drift is the slow gap that opens between what the system was built and tested on and what it now sees day to day, and you catch it by reviewing real decisions regularly, not by waiting for complaints.

Data drift happens when the inputs change: a new product line, a new supplier invoice layout, a seasonal shift in enquiries. The system was never shown these, so its accuracy quietly drops on them. The practical defence is a sample review. Each week or fortnight, a person who knows the work reads a handful of recent decisions and marks them right or wrong. When the wrong ones start clustering around something new, that is drift, and it usually means updating the instructions or examples rather than rebuilding anything.

How should prompt and model changes be handled?

Treat every change to a prompt, a rule or the underlying model as a release: test it against known examples first, record the version, and keep the ability to roll back.

This is where well-run systems and fragile ones part ways. A prompt edited live to fix one complaint can quietly break three other cases. A model update from the provider can change tone, length or judgement overnight. The discipline is simple: keep a set of real past cases with the right answers, run any change against that set before it goes live, and write down what changed and why. It also helps to own the prompts and configuration outright, a point covered in AI vendor lock-in, so a change never depends on someone else's access.

What should happen to the exceptions?

Every exception the system routes to a person should be logged, and someone should review the pile regularly to decide which ones the system should learn to handle and which should always stay with a person.

Exceptions are the best source of improvement you have. A recurring exception is a gap in the instructions. A one-off is just life. Reviewing them together, rather than one at a time as they land, shows the pattern. Some will become new rules or examples. Some will be confirmed as human-only, permanently, and that is a perfectly good answer. Approval gates should loosen only when the record shows they can, never because the team is busy.

Who should own the system after launch?

One named person inside your business owns it, with your builder supporting them, and the owner's name is written into the documentation.

Ownership is where most systems start to fade. The project sponsor moves on, the builder has delivered, and the system becomes nobody's job. Unowned systems are a common thread in why AI projects fail. The owner does not need to be technical. They need to know the work, read the weekly numbers, run or receive the sample review, and decide when something needs changing. Frameworks such as the NIST AI Risk Management Framework and the Australian Government's AI Ethics Principles both put accountability and ongoing monitoring at the centre, and they are worth a read for the owner.

When should you retire an AI system?

Retire it when the job it was hired for has changed or gone, when it no longer beats the baseline, or when keeping it right costs more effort than the work it saves.

That is not failure. Businesses change, and a system built for last year's process can outlive its purpose. The signs are clear if you are watching: falling volume, rising exceptions, a review that keeps finding the same problems. Retiring well means switching it off deliberately, keeping the logs and documentation, handing the work back to a person or a successor, and removing its access to your systems.

Signed at launch, still signed a year later

An AI system earns its place on its hundredth day, not its first. A named owner, a few numbers watched weekly, a regular look at real decisions and controlled changes are what keep it there.

Our builds include a 30-day verdict against the agreed numbers and a handover that sets up this rhythm, as described on the method page. If you have something already live that nobody is quite watching, tell us about it, and we will help you set up the review.

Keep reading