Rollback in five minutes, or it doesn’t count

We ask every client team the same question in the first week of a scale engagement: if this model started giving bad answers at 3pm on a Friday, how long until the previous version is serving again? The honest answers range from “an hour” to “we’d have to ask the vendor”.

Five minutes is the bar.

Not because incidents are frequent, but because a fast rollback changes how the whole organisation treats the system. Risk teams approve releases they can undo. Operators report problems sooner when reporting doesn’t trigger a week of firefighting.

What it takes.

Every model version is an immutable artefact with its evaluation results attached. Serving reads a pointer, not a file. The pointer change is one command, and the command is in the runbook with the name of who may run it.

We drill it.

Before launch, the client team performs a rollback on a staging copy with us watching and not helping. If it takes longer than five minutes, we fix the process, not the people.