Most LLM apps start with the prompt as a string in the code. That works until the prompt starts changing more often than the code around it. From then on, every new word waits for a release.
Why a prompt change becomes a release
A prompt in the code ships with the code. Changing one sentence means a pull request, a review, CI, a build and a deploy. That's the right process for code. For a prompt you adjust after reading a few bad answers, it turns a five-minute fix into an afternoon, and it shuts out the people who often write the best prompts: support leads, product managers, domain experts.
Going back is just as slow. If the new wording makes answers worse, the fix is another release, and until it ships your users keep getting the worse answers.
Four places a prompt can live
Moving the prompt out of the code means choosing where it lives instead.
| Where | Change without a deploy | History and rollback | What you build |
|---|---|---|---|
| In the code | No | Through git, with a deploy | Nothing |
| Environment variables | Usually needs a restart or redeploy | None | Little, but long text is awkward |
| Your own database table | Yes | If you build it | An editor, versions, caching, fallbacks, access control |
| A prompt management service | Yes | Built in | One fetch with a cache and a fallback |
An environment variable is the quickest step, and fine for one short prompt that rarely changes. A database table is where many teams end up, and it works; the effort goes into everything around the table. Wherever the prompt lives, the code that fetches it matters more than the storage, and it's the same code either way.
The runtime pattern: fetch, cache, fall back
Once the prompt lives outside the code, your app reads it while it runs. Done naively, that adds a network call to every LLM request and a new way for your app to fail. Four habits take care of both:
- Cache in memory. Fetch a prompt once, then serve it from memory for a while. A ttl of a minute is a good start; a prompt rarely needs to be fresher.
- Refresh in the background. When the cached copy is older than the ttl, serve it anyway and fetch the new one behind it, so after the first call no request waits on the network.
- Revalidate cheaply. Send back the
ETagyou got asIf-None-Match. The server answers304 Not Modified, with no body, until something changes. It's HTTP's own conditional request, so caches and proxies understand it too. - Time out and fall back. Give each request a few seconds at most and keep the old hard-coded prompt as a fallback. If the prompt store is down, slow, or missing the prompt, your app sends the fallback and logs why.
Here is the whole pattern in TypeScript with plain fetch; the timeout is AbortSignal.timeout, built into Node.js 18 and later. It works against any endpoint that returns a prompt as JSON; this one uses PromptHelm's HTTP API.
const TTL = 60_000; // serve a fetched prompt for a minute
const TIMEOUT = 3_000; // give up on a request after 3 seconds
type Entry = { content: string; etag: string | null; fetchedAt: number };
const cache = new Map<string, Entry>(); // one per process, not per request
async function load(slug: string): Promise<Entry> {
const cached = cache.get(slug);
const response = await fetch(`https://prompthelm.app/v1/prompts/${slug}`, {
headers: {
authorization: `Bearer ${process.env.PROMPTHELM_API_KEY}`,
...(cached?.etag ? { "if-none-match": cached.etag } : {}),
},
signal: AbortSignal.timeout(TIMEOUT),
});
if (response.status === 304 && cached) {
const entry = { ...cached, fetchedAt: Date.now() };
cache.set(slug, entry);
return entry;
}
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const { content } = await response.json();
const entry = { content, etag: response.headers.get("etag"), fetchedAt: Date.now() };
cache.set(slug, entry);
return entry;
}
export async function getPrompt(slug: string, fallback: string): Promise<string> {
const cached = cache.get(slug);
if (cached) {
if (Date.now() - cached.fetchedAt >= TTL) {
// Serve what we have and refresh behind it. A failed refresh waits a full ttl.
load(slug).catch((error) => {
cached.fetchedAt = Date.now();
console.warn(`prompt ${slug}: ${error}`);
});
}
return cached.content;
}
try {
return (await load(slug)).content;
} catch (error) {
console.warn(`prompt ${slug}: served the fallback (${error})`);
return fallback;
}
}Two details to keep in mind. On serverless platforms every instance has its own cache, so a change reaches each one on that instance's next refresh. And two requests that miss the cache at the same moment both fetch; a production version should share the request in flight.
Fill in variables on your server
Most prompts have parts that change with each request: the customer's name, the product, the language. Write them as placeholders, such as {{customer_name}}, in the stored prompt, and fill them in after fetching, in your own process. That has two benefits:
- The values stay with you. The prompt store only ever sees the template. Your users' data doesn't leave your server to reach it.
- A missing value gets caught. If someone adds
{{order_id}}to the prompt before your code passes it, fail and serve the fallback. A literal{{order_id}}sent to a model raises no error; it just produces worse answers.
const VARIABLE = /\{\{\s*([A-Za-z_][A-Za-z0-9_]*)\s*\}\}/g;
export function fill(template: string, vars: Record<string, string>): string {
const missing = [...template.matchAll(VARIABLE)]
.map((match) => match[1])
.filter((name) => !(name in vars));
if (missing.length > 0) throw new Error(`Missing variables: ${missing.join(", ")}`);
return template.replace(VARIABLE, (_, name) => vars[name]);
}How fast a change reaches production
With this pattern, "without a redeploy" means "within one ttl". Each running copy of your app keeps serving the old version until its cached copy is older than the ttl, then picks up the new one. A rollback travels the same way.
- A shorter ttl spreads a change faster and costs more requests. With
ETags, most of those are empty 304 responses. - A longer ttl suits prompts that change weekly and apps that run many copies.
- Whichever you choose, tell the people who publish changes, so they know when to expect them in production.
Doing it with PromptHelm
PromptHelm is this pattern as a service: an editor with variables, versions that never change, one-click rollback, and an SDK that does the fetching for you. The TypeScript SDK caches each prompt for 60 seconds, refreshes it in the background, revalidates with ETags, gives up on a request after 3 seconds, and returns your fallback when anything fails:
import { PromptHelm } from "@prompt-helm/sdk";
// Reads PROMPTHELM_API_KEY from the environment.
const ph = new PromptHelm();
const system = await ph.get("support-reply", {
vars: { company: "Acme", customer_name: user.name },
fallback: SUPPORT_REPLY, // the prompt you had hard-coded
});Other languages use the same HTTP API as the example above. The quickstart walks through it, from signing up to a prompt you change without a deploy. For what happens to a prompt after it's published, read prompt versioning. Calling models through the Vercel AI SDK? See prompt management for the Vercel AI SDK.