Skip to content
Adrythm
Book 20 minutes
AI and automation

Model drift

data drift / model decay / stale model / concept drift

In short

Model drift is an AI system getting worse without anyone changing it. Microsoft names the causes plainly: data distribution changes, training-serving skew, data quality problems, shifts in environments and consumer behavior changes can all make a model stale. NIST sets the bar over the entire lifetime of the system, not at launch.

Software you buy and a model you buy age differently, and Microsoft states the reason directly. Unlike traditional software systems, machine learning system behavior doesn't only depend on rules specified in code, but is also learned from data. Nothing has to be edited for the behavior to change. The inputs change and the behavior follows them.

The causes are listed rather than left vague. Data distribution changes, training-serving skew, data quality problems, shifts in environments, and consumer behavior changes can all cause a model to become stale. And the consequence is stated in business terms rather than technical ones: when a model becomes stale, its performance can degrade to the point that it fails to add business value or starts to cause serious compliance problems.

How drift gets measured is the part that tells you what you are buying. The platform calculates the statistical distribution of the feature's value in the training data. This distribution is the baseline distribution for the feature, and the values arriving in production are compared against it. Built-in monitoring signals for tabular data include data drift, prediction drift, data quality, feature attribution drift, and model performance, and reference data can be historical training, validation, or ground truth data. So monitoring has three separate parts you can ask about by name: a baseline, a signal, and a threshold that raises an alert.

The standards side says the same thing about time. Reliability is a goal for overall correctness of AI system operation under the conditions of expected use and over a given period of time, including the entire lifetime of the system. NIST also names the risk directly, because deployment of AI systems which are inaccurate, unreliable, or poorly generalized to data and settings beyond their training creates and increases negative AI risks. Robustness is a goal for appropriate system functionality in a broad set of conditions and circumstances, including uses of AI systems not initially anticipated.

In practice

Testing at handover tells you the system worked on the day it was handed over. That is worth having and it is not the same as knowing it still works in April. The useful question is not whether a model drifts, because they do. It is who looks afterwards, what they compare against, and how often.

Not the same as

A bug
Nothing broke. The code is what it always was, and the world it reads changed. That is why drift does not show up as an error anywhere.
Hallucination
A hallucination is a confident wrong answer, which can happen on day one. Drift is a decline in how often the answers are right at all.

Why it matters to you

An AI feature is usually bought the way software is bought, once, against a launch date. It behaves more like equipment, which needs somebody assigned to check it on a schedule. Monitoring is either a named line item with a person attached or it does not exist, and the difference only becomes visible months later in complaints nobody connects back to the system.

What to ask or check

  1. 01What is the live behavior compared against, and who holds that baseline?
  2. 02Which signal is watched, and what threshold sends an alert to a human being?
  3. 03Who receives that alert, and what are they expected to do about it?

What people get wrong

That a model which passed testing stays as good as it tested. Its behavior is learned from data rather than only from rules, so a change in what arrives can move it while nobody edits anything.

Red flags

  • An AI proposal with a launch date and no monitoring line.
  • Nobody able to say what the live behavior is compared against.
  • An accuracy figure quoted once, measured before you were a customer.

Who owns it

Whoever runs the system in production. It is a standing duty rather than a handover task, which is why it needs a name attached to it in the agreement.

Where you will see it

Not on any screen. You see it as a slow rise in corrections, escalations or complaints that nobody connects back to the system.

Keep reading

Hallucination

A hallucination is AI output that sounds right and is not. OWASP describes the model filling gaps in its training data using statistical patterns, without understanding the content, so the answer can be fluent and unfounded at once. Its own first example is an airline that was successfully sued over its chatbot.

AI agent

An AI agent is a model that has been granted the ability to take actions, not just produce text. OWASP says the damage one can do comes from three grants: excessive functionality, excessive permissions and excessive autonomy. What it is allowed to do matters more than how good it is.

Definition of done

A definition of done is the written list of conditions every piece of work must meet before anyone calls it finished. The Scrum Guide treats it as a gate: work that misses it cannot be released or even shown at the review. Agile Alliance warns that an unwritten one loses most of its value.

Training data

Training data is what a model learned from. The question owners ask is whether their own data joins it, and there is no single answer: OWASP names three separate stages, pre-training, fine-tuning and embedding. A commitment worth having names the stage and comes in writing.

Retrieval augmented generation

Retrieval augmented generation is how an AI answers from your documents without being trained on them. OWASP describes it as combining a pre-trained model with external knowledge sources at answer time. So the documents sit in a store the system reads from, and who can read that store is the question.

Prompt injection

Prompt injection is when text an AI system reads becomes an instruction it follows. OWASP notes the text does not have to be visible to a person, only parsed by the model, and that no fool-proof prevention is known. The UK's national cyber security body says the same.

Want this explained against your own numbers?

Twenty minutes, a straight answer, and no follow-up sequence if you decide not to work with us.

Book 20 minutes