Your support bot keeps producing an awkward line. "What do you feel might collapse there?" shows up in conversation after conversation. You add one line to the system prompt: "Do not ask what might collapse." The next week a different stock phrase appears. You add another line. A few weeks later the prompt carries more than twenty prohibitions, and some of them contradict each other. This post is about not going down that road.
Where should a rule live when the model breaks it?
Don't patch a model's mistakes with prohibitions; split the rules across three places: principles ("what and why") in the prompt, invariants that hold regardless of context in code, and context-dependent quality in an evaluation set.
Each place is good at something different. The prompt gives the model a direction to judge by. Code reliably blocks what can be checked mechanically, without judgment. An evaluation set re-measures, on every change, whether the output is good in a human's eyes. Asking one place to do another's job gets expensive.
Why not just add another "don't" line?
A prohibition tells the model which path to avoid but not which path to take. Smaller models in particular simply move to a different stock phrase next to the blocked one. In one conversational product, banning one awkward question just let a similarly shaped question take its place, and the patched-on rules passed twenty within a few weeks and began to conflict (2026-10, counted while reviewing real on-device conversations).
There is a second trap: models copy example sentences from the prompt verbatim. Write "e.g. 'How did that make you feel?'" to explain a rule, and that exact sentence turns up in conversations. So when you fix the prompt, do two things:
- Delete the prohibition list and rewrite the purpose those prohibitions protected as a few principles ("Don't ask again about what the user already said — the point is to show you listened")
- Keep only three or four examples that differ a lot from each other, wrap them in tags, and say they show the tone, not sentences to reuse
This matches the model vendor's own guidance. Anthropic's prompting guide says to tell the model what to do instead of what not to do, and to wrap a few diverse examples in tags.
How do you decide what code should enforce?
"Then why not check everything in code and force a rewrite?" That road is blocked too. If context-specific quality rules ("don't press someone worried about the future for concrete scenes") also get filtered and rewritten by code, the rules grow without end again, and every caught turn is slower by one rewrite call — measured at 1–2 seconds per turn.
There is one test: can you decide true or false without knowing the context? Things that belong in code:
- Safety rules — must never break in any situation
- Agreement between what the model says and server state — if it says "tap the button below," the button must be on screen
- Example copying — if output overlaps a prompt example by N or more characters, rewrite
- Repeated questions — nearly the same question as recent turns
- Format and length limits — are all JSON fields present, is it not too long
Anything that is right in one context and wrong in another is measured by an evaluation set (a fixed bundle of a few dozen representative conversations you re-run on every change), not by code. A human judges whether the conversation was good and records pass/fail with a one-line reason.
How should you ask for a rewrite when a check fails?
Asking for a whole new output when a check fails creates new problems. While rewriting, the model changes other fields that were fine, those get caught, and the record ends up saved with empty fields. In one structured output, the interpretation and quotation fields were emptied exactly this way. Fix it in this order:
- In the rewrite request, state which field failed and why, and tell it to keep the other fields
- If both attempts fail, merge the fields that passed from each response
- Count what percentage of rewrites each rule causes. A style rule that triggers more than 20% of the time produces only latency and empty fields — remove or loosen it (in one case, a 40-character sentence limit became 80)
Classifiers work the same way. Patching false positives ("it flagged this as risky too") with "this is also not a case" examples just lets the next topic trip it. Instead, rewrite the criterion as whether a verifiable fact is present, require positive verdicts to quote the evidence verbatim from the source, and have code check the quote against the source and reject the verdict if it isn't there — a principle (prompt) paired with an invariant (code).
What to paste into your prompt workflow
Give this instruction to whoever (or whichever agent) maintains the prompt:
When model output breaks a rule, check these in order before adding a prohibition.
1. Can this rule be judged true or false without knowing the context?
- Yes -> make it a code check. On failure, say which field failed and why, and rewrite only that field.
- No -> revise the prompt's principles (what and why). Add this case to the evaluation set.
2. How many prohibitions ("don't ...") does the prompt have? Above 5, rewrite them as purpose-based principles.
3. Does the output overlap examples in the prompt? If so, cut the examples and make them more different from each other.
4. Measure what percentage of rewrites each code check causes. Above 20%, remove or loosen the rule.
Checklist
- You have counted the prohibitions left in the prompt
- Every example differs enough from the others, and code measures output overlap with them
- The code-check list contains no rule that needs context to judge
- Rewrite requests fix only the failed field
- You track each rule's rewrite rate and the latency it adds
- Context-dependent quality is judged by humans in an evaluation set, re-run on every change