Skip to content

Guides

A/B testing LLM prompts without false winners

Checking a prompt test as results arrive finds winners that are not there. How always-valid intervals, split checks and guardrails keep the verdict honest.

13 min readUpdated 10 November 2026

You rewrote the support prompt. Two days later the test dashboard says the new version resolves four points more tickets, with a 95% interval that doesn't touch zero. You ship it. A month later the resolution rate is where it always was.

Nothing was wrong with the arithmetic. What went wrong was when it was read. This guide covers the few decisions that make a prompt A/B test tell the truth: what you randomize, what you measure, how to read a result while the test is still running, and how to notice when the test itself is broken. The numbers come from simulations run with the same statistics PromptHelm uses, and the methods from the papers linked along the way.

What a prompt A/B test compares

Two versions of a prompt, the text and the model settings that go with it, serve the same traffic over the same days. Comparing this week's prompt with last week's isn't a test: your users, their questions and even the model behind the API change from week to week, and a before-and-after comparison can't tell those changes from yours.

Randomize by user or conversation, not by call. A user who gets the formal prompt for one reply and the casual one for the next has a worse experience than either would give, and their outcomes aren't independent anyway. Hash the unit together with the test's id into a bucket from 0 to 99, and serve the new version to buckets below its share. The same user always gets the same version, there's no table of assignments to keep, and raising the share from 20% to 50% only moves users from live to new, never back.

1 · A call arrives

request

For a user or conversation: the unit, here user-1042.

2 · Assign

request

hash(test id + unit) → bucket 0–99. The same unit always lands in the same bucket.

Bucket below 20

New version, v5: 20% of units.

Bucket 20 or above

Live version, v4: the other 80%.

3 · Measure

request

Your code reports how the call went: the user kept the reply → 1, or a score from 0 to 1.

4 · Add up, per version

analysis

n, Σx and Σx², with tokens and latency beside them. Totals are all the analysis needs.

5 · Decide

analysis

An always-valid interval for the difference and a check of the split: ahead, behind, the same, or not yet clear.

One call's way through a test. The first three steps run in the request; the last two only need totals.

Decide what better means before you start

Pick one primary metric, an outcome you can record for each call. Most fit one of two shapes:

  • Success, yes or no: the user kept the reply, the ticket closed without a person, the JSON parsed, the tool call worked.
  • A score from 0 to 1: a rubric, a rating, or a grade from another model.

Choose it before you look at any data. Picking whichever metric looks best afterwards is the same mistake as peeking, in a different form. Watch tokens, latency and errors beside it as guardrails rather than goals: a prompt that resolves two points more tickets with twice the tokens may not be better.

Then write down the smallest difference that matters, say five points on a 30% success rate. It decides how long the test runs, and below, how its interval is tuned. Some outcomes arrive late, such as a ticket that closes tomorrow; keep an id for each call so the outcome can find its way back to the version that caused it.

Why looking early finds winners that aren't there

A 95% interval makes one promise: when there's no difference, it excludes zero only 5% of the time. That promise holds for a single look at a sample size fixed in advance. A live dashboard recomputes the interval after every batch of results, and each look is a fresh chance for noise to cross the line. The chances add up.

To see how much, we simulated 4,000 A/A tests, where both versions are the same prompt at 30% success, and looked after every 100 outcomes per version. A test "found" a difference if its interval ever excluded zero:

A/A tests that ended up showing a difference

Fixed 95% interval, checked at every lookAlways-valid 95% interval
Share of 4,000 A/A tests in which each interval excluded zero at least once
LooksFalse alarms
1 look
Fixed interval: 4.4%
Always-valid interval: 0.1%
5 looks
Fixed interval: 13.7%
Always-valid interval: 0.4%
10 looks
Fixed interval: 19.1%
Always-valid interval: 0.8%
30 looks
Fixed interval: 28.4%
Always-valid interval: 1.5%
100 looks
Fixed interval: 38.0%
Always-valid interval: 2.2%

Dashed line: the 5% a 95% interval promises

Simulated with the statistics code PromptHelm runs. One look keeps its promise; ten looks find a winner in one test in five.

Armitage, McPherson and Rowe measured the same effect for clinical trials in 1969, and it never stops: look often enough and a fixed interval eventually excludes zero in almost every A/A test. Prompt tests invite it more than most. The dashboard updates live, and the new prompt was written to win.

There are two honest ways out. Fix the number of outcomes in advance and look once, or use an interval built to be looked at.

Intervals that stay honest at every look

The mixture sequential probability ratio test (mSPRT) asks a different question. Instead of "is the difference non-zero at this sample size?", it keeps a running likelihood ratio: how much more likely the data are under some real difference than under none, averaged over the differences you'd plausibly see. When there's no difference, that ratio is a martingale, and the chance that it ever climbs above 1/α is at most α, however often you check (Robbins, 1970). Johari, Koomen, Pekelis and Walsh turned this into p-values and intervals for A/B tests that stay valid at every look.

With a normal approximation, the 95% always-valid interval for the difference is:

diff ± r,  where

  r = √( V·(V + τ²)/τ² · [ 2·ln(1/α) + ln((V + τ²)/V) ] )
  V = s²(A)/n(A) + s²(B)/n(B)

diff  the new version's mean outcome minus live's
s²    each version's sample variance, n its outcomes
τ     the smallest difference that matters, e.g. 0.05
α     0.05 for a 95% interval

τ sets the size of difference the test is most sensitive to, so set it to the smallest difference that matters. Early on, the interval is much wider than the usual one, and it narrows as outcomes arrive. Here is the same A/A run read both ways:

An A/A test, looked at after every 100 outcomes

Both versions are the same prompt, at 30% success. Any lead is noise.

The new version's leadFixed 95% intervalAlways-valid 95% interval

Fixed 95% interval

No difference

−0.5 to +3.6 points

Always-valid 95% interval

No difference yet

−1.6 to +4.7 points

4,000 outcomes per version · 40 looks · the fixed interval raised a false alarm at 7 of them

A real simulated run, seeded so it replays the same. The fixed interval names a winner at seven looks between 1,500 and 2,200 outcomes, then the lead fades; the always-valid interval keeps zero inside throughout.

In the 4,000 A/A tests above, it raised a false alarm in 0.8% of them after 10 looks and 2.2% after 100: under its 5% at every look. The price is data. For a real lift from 30% to 35% it found the difference after a median of 1,400 outcomes per version, and by 2,400 in 80% of tests. A fixed test sized for 80% power needs about 1,400, but only if you look once. Roughly, you pay 1.7 times the data for the right to look whenever you like and to stop as soon as the answer is clear, which for a large difference comes early.

How many outcomes you need

For a 30% success rate, split 50/50, these are the outcomes per version after which the always-valid interval is narrower than the difference you care about, the point at which the test can say whether that difference is there:

Smallest difference that mattersOutcomes per versionCalls in all
10 points (30% → 40%)about 430about 850
5 points (30% → 35%)about 1,600about 3,250
2 points (30% → 32%)about 9,900about 19,900

Halving the difference roughly quadruples the data. Giving the new version a smaller share limits the harm a bad prompt can do while you learn, but the test takes longer: at 20%, the new version needs about 1,050 outcomes for a five-point difference while live collects four times as many, about 5,200 calls in all against 3,250 at 50/50. Divide by the outcomes you get a day for the test's length, and run whole weeks if your traffic differs by weekday.

Check the split before you trust the result

If you asked for 50/50 and got 10,300 calls on live and 9,700 on the new version, the split itself says something is broken. A chi-square test measures how surprising that is: here χ² = 18, and p ≈ 0.00002. This is a sample ratio mismatch, and Fabijan et al. catalogue its causes in experiments at scale. In LLM apps, a few are common:

  • The new prompt is longer and times out more. Calls that time out never report an outcome, so the new version quietly loses its worst calls.
  • One of your deployments runs an older client that doesn't know about the test and sends every call to live.
  • Calls that can't use the new version, such as one missing a variable it needs, fall back to live and are counted there.
  • A cache or retry layer that only some calls pass through.

Check the split all the time, with a strict threshold such as p < 0.001 because you check it so often. When it trips, stop reading the metric and find the cause. A mismatch doesn't say which version is better; it says the comparison isn't fair.

Stop early on bad news; ship on purpose

Let a test stop itself when the new version is clearly worse, its interval entirely below zero. Ending a test sends every call back to the live version: cheap, instantly undone, and it limits the damage a bad prompt can do while nobody is watching.

Shipping is different. Don't let a test ship a winner by itself. The metric doesn't see tone, the odd edge case or the guardrails, and someone should read both versions' answers before the new one becomes the only one.

  • Change the share, start the count again. If the new version had 10% on Monday and 50% by Friday, a pooled comparison weighs its days differently from live's. When traffic differs by day, that alone can move the result, or even flip it (Simpson's paradox). Compare only outcomes collected at the current share.
  • Expect less than the test said. The difference a test reports when it first crosses the line is, on average, bigger than the real one: tests that happen to run high get there first. Report the interval, not only the estimate.
  • One test per prompt at a time. Two overlapping tests on the same prompt share their users and muddle each other.

A checklist for your next prompt test

  • Randomize by user or conversation, by hash, and keep the share steady.
  • One primary metric, chosen before the test; tokens and latency as guardrails.
  • Write down the smallest difference that matters, and estimate the test's length.
  • Read an always-valid interval if anyone will look before the end, or fix the sample size and look once.
  • Check the split continuously, and stop reading the metric when it's off.
  • Let the test stop on clear harm; ship by hand, after reading real answers.

How PromptHelm runs prompt A/B tests

PromptHelm serves your prompts at runtime, so a test is a change on the prompt's page, not a deploy:

  • A test compares the live version with one new version: your draft, saved as a version, or an earlier one. Each brings its own model settings, so a test can compare two models as well as two texts.
  • The SDK picks the version for each call on your server, from a hash of the unit you pass. The unit never leaves your server, and neither does any text: the SDK sends totals.
  • The prompt's page shows the always-valid 95% interval, tuned to your smallest difference, and says whether the new version is ahead, behind, the same, or not yet clear, and roughly how many more outcomes it needs. It waits for 100 outcomes per version, flags a mismatched split, and compares only outcomes at the current share.
  • A guarded test stops itself if the new version is clearly worse. Nothing ships by itself: making the new version live is one click, and you get an email when there's an answer.
app.ts
const r = await ph.resolve("support-reply", {
  vars: { company: "Acme", customer_name: user.name },
  fallback: SUPPORT_REPLY,
  unit: user.id, // the same user always gets the same version
});

const reply = await openai.responses.create({ model: r.config.model ?? "gpt-5", input: r.text });
r.usage(reply.usage); // tokens, and the call's latency since resolve
r.outcome(userKeptTheReply); // true or false, or a score from 0 to 1

// An outcome that comes later, from any process: keep r.ticket, then
// ph.outcome(ticket, true)

On the Pro plan

A/B tests are part of Pro. The quickstart's A/B tests section shows the options, from force to onResolve.

Questions

Can I just use a t-test?

Yes, if you decide the number of outcomes per version before the test starts and look at the result once, when it has them. A t-test or a fixed 95% interval is then valid, and it needs fewer outcomes than an always-valid interval. The always-valid one is for the usual case, where people look at a dashboard while the test runs.

What about multi-armed bandits?

A bandit moves traffic toward whichever version is ahead as it learns. That earns more during the test when outcomes arrive within seconds, but it makes a clean estimate of the difference harder, and many prompt outcomes arrive late: a ticket closes tomorrow, a user comes back next week. To decide whether to ship a prompt, a fixed share with an always-valid interval is simpler to trust.

Should a model-graded score be the metric?

It can be: a score from 0 to 1 works with the same interval. Check the grader against a sample of human judgements first, and keep its prompt and model fixed for the whole test, or you're testing two things at once.

How long should a prompt test run?

Until it has an answer, and for at least a full week if your traffic differs by weekday, so both versions see the same mix of days. Divide the outcomes it needs by the outcomes you get a day for a first estimate.