GenAI systems in production: defining the conditions for suspension, rollback and escalation
For a GenAI system, plan triggers, a suspension capability, a reference version, an escalation chain and tested resumption criteria.
A rollback plan for a GenAI system is not just about putting an old model back online. The team must know which capability to suspend, for which users or data, which reference version to restore, who decides on escalation and which evidence authorises resumption. Without these decisions prepared in advance, a technical incident becomes a series of improvised actions that can widen the scope of the problem.
In short
- Distinguish three decisions: suspending a capability, reverting to a reference version and escalating to the competent owner.
- Define observable triggers by type of deviation: access, data, quality, tool action, availability, cost or traceability.
- Version the configuration actually served, not just the code: model, prompts, rules, index, sources, tools, rights and test results.
- Test the restriction and resumption path. A runbook that has never been rehearsed does not prove the team can scale the system back without improvising.
This method helps prepare the operation of a GenAI system. It sets no universal threshold, no recovery time and no acceptable risk level. These decisions depend on the task, the data, the users, the connected tools and the organisation's responsibilities.
Why are suspension, rollback and escalation three different decisions?
The same signal does not always call for the same response. A rights leak, a quality regression after a model change and a vendor outage can all lead to a service interruption. Yet they are handled neither with the same evidence nor by the same owner.
| Decision | Question to answer | Intended effect |
|---|---|---|
| Suspension | Which capability must stop being available now? | Reduce exposure during the analysis |
| Rollback | Which reference state can be restored, on which scope? | Return to a known, tested configuration |
| Escalation | Who must qualify the impact, authorise an exception or decide on resumption? | Place the decision at the right level of responsibility |
Suspension can apply to a write action, a connector, a corpus, a user population or an entire journey. It does not assume that a cause has already been established. Rollback requires an explicit reference and a check that restoring it does not reintroduce another risk. Escalation organises the decisions that go beyond the team monitoring the system.
NIST's Cybersecurity Framework 2.0 separates incident response activities from those that restore affected assets and operations. This distinction provides a useful grid for a GenAI system, without turning the procedure presented here into a compliance requirement: containing and deciding do not yet mean restoring.
Which triggers should you write down before opening?
A useful trigger describes a signal, the scope concerned, the first possible restriction and the person who then takes over. Avoid thresholds copied from another system. A wrong answer on a search suggestion and an unauthorised write action in a business tool do not have the same consequences.
| Observed signal | First measure to prepare | Decision to escalate |
|---|---|---|
| A role retrieves an unauthorised source or action | Suspend the capability or scope concerned, keep the test and trace evidence | Security, data or authorisation owner |
| A candidate version degrades a critical case | Revert to the reference version or limit the exposed population | Technical lead and business owner of the use case |
| A tool action produces an unexpected or unexplainable effect | Disable the impactful action and keep only read mode if that remains acceptable | Tool owner, operations and the business concerned |
| A corpus source becomes doubtful, outdated or corrupted | Isolate the source or associated index, then rerun the relevant retrieval cases | Source owner and system owner |
| An external dependency stops responding or no longer behaves as expected | Switch to the planned degraded mode, throttle the flow or suspend the journey | Operations and service owner |
| The traces needed for diagnosis are missing or inconsistent | Freeze any widening of the scope and preserve the available elements | Technical lead, security or governance depending on the context |
The trigger should not try to name every possible cause. It must enable a first safe and reversible action. The decision to extend, maintain or resume the service comes next, with the evidence suited to the case.
NIST's generative AI profile describes a risk management approach to adapt to the system's objectives, context and impacts. In a GenAI runbook, this adaptation consists in particular of linking each trigger to a scope, a restriction action, an owner and a resumption condition.
Which version must you be able to restore?
A reference must not be defined as "the last version that seemed to work". It must describe the state actually exposed to users and the evidence that led to accepting it.
Keep a reference manifest that includes at least:
- the application version and the relevant deployment parameters;
- the model, its endpoint or variant, and the parameters that change its behaviour;
- the active prompts, output rules, guardrails and routing rules;
- the indexes, corpus, source versions and retrieval rules used;
- the connected tools, the enabled capabilities and the associated authorisation rules;
- the test sets, results and known limits that accompanied the go-live decision;
- the owner of the reference and the procedure to restore or withdraw it.
This list does not impose a single format. It prevents a rollback from restoring the code while leaving an index, a prompt, a tool or an access rule in the state that produced the deviation.
For a document source, the reference must also state which content and which versions are served. Our guide to versioning, updating and withdrawing a source document in a RAG details how to tie the index, chunks, rights and withdrawal evidence to this change cycle.
When is a rollback not enough?
A rollback addresses an identified change. It does not automatically repair the effects of undue access, a business action already executed or data exposed in an unplanned channel.
| Situation | Why rollback alone is not enough | Complement to plan |
|---|---|---|
| Access right removed or wrongly applied | Restoring an old model does not necessarily block current access | Scope suspension, revocation or rule fix, negative tests |
| External action already triggered | The reference version does not undo the effect in the business tool | Stop the action, appropriate business handling and execution traceability |
| Erroneous knowledge source | Restoring the code can keep the same corpus or index | Isolate the source, return to a known corpus version and run retrieval tests |
| Vendor dependency incident | Reverting to a previous state can restore the same dependency | Degraded mode, throttling, change of journey or wait for a decision |
| Insufficient diagnostic traces | A fast resumption can erase the context needed for analysis | Keep the authorised elements, reconstruct the versions and review before resuming |
The goal is not to keep all data or all traces indefinitely. It is to make an informed decision possible, while respecting the applicable security, confidentiality and retention rules.
NIST SP 800-61 Rev. 3, published in April 2025, places preparation, detection, response and recovery within a risk management approach. It points out that the effectiveness of the response also depends on coordination, analysis and recovery, not only on an isolated technical action.
Which escalation path should you define?
An escalation path is not a contact directory. It states who can take which decision in the time available, with which information. An organisation can adapt the following matrix to its own roles.
| Moment | Expected decision | Role that must be identifiable |
|---|---|---|
| Detection | Qualify the signal, protect the useful evidence and choose the first restriction | Operations team or service owner |
| Containment | Reduce the scope without increasing exposure | Technical lead with the relevant access or tool owners |
| Impact assessment | Determine the users, data, tools and decisions affected | Business, security, data or compliance depending on the case |
| Return to reference or corrective change | Validate the state to restore and the tests to rerun | Technical owner and scope decision-maker |
| Resumption | Authorise reopening or maintain a restriction | Service owner with the owners concerned |
| Review | Document the deviation and improve the runbook | Owners who executed or approved the response |
Prepare the escalation sheet with concrete elements: service concerned, enabled capabilities, reference version, scope currently suspended, actions already taken, data or tools potentially affected, tests to rerun and the decision requested. Avoid circulating sensitive business content or secrets in an incident channel that is not designed to receive them.
How do you check that the procedure works before an incident?
A short exercise is worth more than an untested runbook. It can rely on simulated data and actions, without causing any real effect in a business system. Aim to check the team's ability to act, not to demonstrate perfect availability.
- Targeted suspension. The team disables a specific capability, for example a write action or a connector, then checks that the other authorised journeys remain within the planned scope.
- Return to the reference. It identifies the approved manifest, restores the chosen configuration and reruns a representative test set.
- Revocation or restriction. It simulates a rights change or a source withdrawal and checks the refusals, the traces and the absence of undue retrieval.
- Timed escalation. It prepares the summary needed for a decision without setting a universal deadline: signal, reach, measures already taken, version, options and recommendation.
- Conditional resumption. It reopens the scope only after the tests and decisions defined for the scenario.
The guide on handing over a GenAI system to internal teams already proposes a takeover exercise to check that the team can restrict a system, prepare a change and revert to a reference. This page adds the suspension decision, the escalation conditions and the criteria that avoid a resumption based on an impression.
Which resumption criteria should be made explicit?
Resumption must not be the automatic consequence of an alert disappearing. Define what must be true for the scope concerned:
- the cause, or the hypothesised cause, is qualified well enough to decide;
- the restored or corrected version is identified and its differences are readable;
- the roles, data, tools or actions concerned have been tested;
- the critical cases and planned refusal cases match the expected behaviour;
- the traces needed for operations are available again;
- the owners concerned have accepted the resumption, the remaining restriction or the follow-up plan.
This list is not an automatic green light. It distinguishes a decided resumption from a forced return to service. The go/no-go framework between an LLM prototype and production helps tie these criteria to the scope, the evidence and the owners who accepted the opening.
FAQ
Should you stop the whole system when a deviation is detected?
Not necessarily. Prepare levels of restriction: disable an action, isolate a source, restrict a population or suspend an entire journey. The right level depends on the possible impact and on the ability to show that the rest of the system remains within its authorised scope.
Does a rollback always mean restoring an old model?
No. The reference can concern a model, but also a prompt, an index, a corpus, an access rule, a connector or a combination of these. The decision must restore the state that is known and testable for the scope concerned.
Which recovery time should you promise?
There is no universal recovery time. It depends on the nature of the deviation, the data, the tools, the authorisation rules, the architecture and the decisions required. Set an objective suited to the service and communicate the limits rather than promise an unverified recovery.
Who authorises resumption after a suspension?
It depends on the scope affected. Prepare the matrix before the incident: operations to observe and restrict, the technical owner to restore and test, then the business, data, security or compliance owners when their scope is concerned. The person who authorises resumption must be able to read the evidence and the remaining limits.
Methodology and sources
- NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, July 2024, consulted on 13 October 2026. The profile presents voluntary, context-specific governance, mapping, measurement and risk management actions for generative AI.
- NIST, SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management, April 2025, consulted on 13 October 2026. The publication links preparation, detection, response and recovery to risk management activities.
- NIST, Cybersecurity Framework 2.0, consulted on 13 October 2026. The framework distinguishes, among other things, response, recovery, communication and continuous improvement activities, to be adapted to the context.
This page is an operations method for GenAI systems. It is neither legal advice, nor a complete incident procedure, nor a guarantee of availability, security or compliance. Thresholds, owners, escalation channels and resumption conditions must be validated for the organisation's context.
Do you need to prepare the suspension, rollback and escalation of a GenAI system before opening it to production? Describe the scope to IgnitionAI. A first conversation can help turn signals, reference versions, tests and responsibilities into a workable procedure.