What a reproducible AI evidence test looks like, step by step.

A reproducible AI evidence test is one where someone else can take your logged inputs, rerun or inspect the run, and reach the same conclusion within an agreed tolerance. Tolerance is the amount of difference between two runs that you agree in advance is still acceptable for the decision the output supports.

This guide is for compliance, risk and data leads in regulated organisations. You do not need identical wording from the model each time. You need a record strong enough that the decision it supported can be checked.

Three kinds of reproducibility and which one your test needs

The Association for Computing Machinery (ACM) separates repeatability, reproducibility and replicability. For AI-assisted work, three practical levels are more useful:

• Exact output reproduction. Identical text or labels. This is rarely realistic with a hosted language model.
• Procedural reproducibility. Another team can follow the documented workflow. This is realistic if you keep a full input bundle.
• Result reproducibility. Another team reaches the same finding within tolerance, even if the wording differs. This is the usual target.

Procedural reproducibility is the floor. Result reproducibility sits on top, judged against the tolerance you set in advance. Name the level in the test plan before the first run.

Why the same prompt can give a different answer

A language model does not look up an answer. It samples one. Each token is drawn from a probability distribution shaped by the input and the decoding settings. Temperature, top-p and top-k all change how varied the output is.

Hidden inputs add more variation. Many tools add system instructions you never see. A chat window also carries earlier conversation into each new answer. Two people typing the same prompt may send quite different requests.

The serving software adds its own noise. Thinking Machines Lab, in its paper Defeating Nondeterminism in LLM Inference, shows why. Shared servers batch your request with other users' requests. Batch size changes the order of floating-point sums, so the same prompt can produce different tokens even at temperature 0.

Providers can also update or retire a model behind the same name.

So does temperature zero make output reproducible? No. It reduces variation within one session. It does not prove the run can be repeated on another day. State every setting you cannot see or control as a limitation in the record.

Which rules expect this kind of AI evidence

This is general information, not legal advice. Confirm scope with counsel.

SR 26-2 and US banking model risk guidance

For the wider picture of what each US and EU rule asks of a financial firm, see the AI regulations for financial services guide.

On 17 April 2026 the Federal Reserve, the OCC and the FDIC issued revised guidance on model risk management. The Federal Reserve published it as SR 26-2. It supersedes SR 11-7 from 2011 and SR 21-8, the 2021 statement on BSA/AML systems. SR 11-7 is no longer current guidance.

OCC Bulletin 2026-13 says generative AI and agentic AI models are not within the scope of the guidance. Out of scope does not mean no questions. Supervisors and auditors still ask how a decision was made.

The NIST Generative AI Profile

NIST published the Generative Artificial Intelligence Profile (NIST AI 600-1) in July 2024. It is voluntary and cross-sector. Its primary considerations include governance, content provenance, pre-deployment testing and incident disclosure. Testing means little if nobody can repeat it. Incident disclosure needs logs of what the model received and returned.

EU AI Act and GDPR

Regulation (EU) 2026/1744 amended the EU AI Act (Regulation (EU) 2024/1689). Under it, high-risk obligations for Annex III systems apply from 2 December 2027. For high-risk systems under Annex I, linked to regulated products, they apply from 2 August 2028. Article 15 of the AI Act requires high-risk systems to reach an appropriate level of accuracy and cybersecurity, and to perform consistently throughout their lifecycle.

Article 32(1)(d) of the GDPR requires a process for regularly testing, assessing and evaluating security measures.

Reviewed through 26 October 2026.

Step 1: write the test claim and acceptance criteria

"The model works well on complaints" is a weak claim, because nobody can fail it. A strong claim reads: "The model assigns each complaint to one of five categories with at least X agreement with the reference set." You set X. Now the test has a number to beat.

The right number depends on use. A score that is fine for routing complaints may be unacceptable for flagging fraud cases. Before any run, write down the intended use and the decision the output affects. Define the ground truth. Set the error threshold. List the high-risk edge cases and the consequences of failure.

Scale effort to materiality. A tool that rewords letters needs light testing. A model that feeds credit decisions needs every step.

Sign and date the criteria before the first run. A tolerance chosen after seeing results proves nothing, and the dates will show it.

Step 2: freeze the input bundle

The input bundle is everything another reviewer needs to recreate your run. The prompt is only one part of it. Record these items as they were on the day:

• AI system: provider, the most specific model identifier or snapshot, and the access method.
• Execution context: dates, system instructions, tools and any conversation context.
• Inputs: exact prompts, templates, input data and preprocessing steps.
• Configuration: temperature, top-p, seed and output limits.
• Environment: library versions, hardware and container image.

Version prompts like code. Give each one a file, a version number and a change note. A one-word edit can shift labels across a whole dataset.

Store the bundle with a hash, so any later change to a file shows up at once.

Step 3: run the test in a controlled environment

A test in a personal chat window inherits earlier turns. Nobody can rebuild what the model saw. Run every test call through an API with fixed settings, from a clean job with no history.

Use an isolated container or sandbox job. Tools such as DVC, MLflow and a pinned Docker image help.

Each run should log:

• The request, with timestamp and model identifier
• The full raw response, before any cleaning
• Every tool and API call the model made
• Errors, timeouts and retries

Watch for silent retries and fallbacks. Some services switch to another model when the first is busy. Log the model that actually answered.

If a call fails, log the failure as a result. Never rerun it quietly until it succeeds.

Step 4: repeat the runs and measure the spread

A single run tells you almost nothing about reliability. Run the same input bundle several times. Report the mean and the spread, not the best run. Picking the best run hides the variation a reviewer needs to see.

Raw agreement can flatter a model. If most items sit in one category, runs will often match by chance. A chance-corrected statistic such as Cohen's kappa is more informative for labels.

Look past the wording. Two outputs can differ in phrasing yet support the same decision. Two can look almost identical yet flip a risk rating. When stakes are high, review changed items one by one.

Compare the spread with the tolerance you signed in Step 1. If it sits outside, record a fail.

Step 5: check the answers against a reference set

A model can give the same wrong answer every time. Consistency proves stability, not correctness. Only a check against known correct answers shows the model is right.

Build the reference set independently of the model, labelled by people. Write a rubric that defines each label. Use an odd number of reviewers so ties resolve cleanly. Fluent text can persuade reviewers, so score answers against the rubric, not against how confident they sound.

Choose metrics that suit the task. When classes are imbalanced, accuracy hides errors. Precision, recall and F1 show whether the model misses rare but important cases. Include hard cases near the edge of what the model can do. That is where it is most likely to fail.

Step 6: package and sign the evidence record

The evidence record is what a reviewer opens when someone asks for the basis of a decision. It should hold:

• The input bundle from Step 2
• Raw outputs and error logs from every run
• Human corrections, each with its reason
• Evaluation scores compared with the tolerance
• Analysis code and the final dataset
• A limitations note for settings nobody could see or control

Store it in write-once or hashed form, with access controls. A record anyone can edit proves nothing. For the fields to capture, see this guide to what regulated organisations must log for AI audit evidence.

Archived outputs outlive the model. Providers retire models, so a rerun may become impossible. The stored outputs still let a reviewer rebuild the finding.

Separate what the model produced from what a person changed. Without that, the record cannot show who made the decision. The signed record then takes its place as one row in the AI evidence register, beside the system's approvals and changes.

Re-test after model changes

A passed test describes one model version on one date. A silent provider update can void it.

These events should trigger a re-test:

• A provider model update or retirement
• A prompt or template change
• A new data source or preprocessing step
• A library or hardware change
• Drift alerts from production monitoring

Link each new record to the previous one. A reviewer should see which version was approved, what changed and why the next version passed. Keep the last approved input bundle ready, so a regression test can run the day a change is announced.

Common mistakes that break an AI evidence test

Most failed tests fail on missing records, not bad models. Watch for these:

• Saving only the prompt, without settings, model identifier or data
• Assuming temperature 0 guarantees identical output
• Treating a model name as a preserved model
• Reporting the best run instead of the spread
• Choosing the tolerance after seeing results
• Calling AI disclosure reproducibility

The last one is subtle. Saying "AI was used" is honest, but it gives a reviewer nothing to rerun. Disclosure says a model was involved. Reproducibility lets someone check what it did.

Before sign-off, ask a colleague who did not run the test to rebuild one result from the record alone. If they cannot, the record is not finished.

Next step

The free Handvantage handbook sets out the records and templates for this kind of test. No email is needed. Get it at workspace.handvantage.com/handbook.