System prompt too long? What to do instead of adding more rules

Yes, a system prompt can be too long. Every line reaches every request, so a rule written for one edge case competes with everything else the model is trying to follow, even in conversations where that rule doesn’t matter. The fix is to keep only what applies to every conversation and move the rest somewhere more reliable.

This guide is for teams running an AI agent in production whose prompt grew one incident at a time.

Can a system prompt be too long?

Yes, and the limit that matters arrives long before the model’s context window fills up. There is no universal number, but the evidence points one way:

• Following instructions gets harder as they pile up. In the IFScale benchmark, the best frontier models reached only 68% accuracy at 500 instructions, and models favored instructions that appeared earlier. The test used simple keyword rules in a report-writing task, so treat it as a direction and not a threshold for your agent.
• Anthropic gives the same advice for agents. Its context engineering guide says to find “the smallest set of high-signal tokens” that gets the outcome you want. It also warns that hardcoding brittle logic in prompts “creates fragility and increases maintenance complexity over time.”
• Coding tools report the same pattern. When a line in CLAUDE.md keeps getting ignored, Claude Code’s best practices offer two explanations: “Either the file has grown long enough that the line gets lost, or the line is a rule that can’t have exceptions, in which case it belongs in a hook.”

Length also has a direct cost, because every token in the prompt is sent on every request.

How do you know yours has grown too far?

Any of these is a good sign:

• A fix for one workflow changes behavior in another.
• Nobody can say why a line is there or whether it still helps.
• You added IMPORTANT, ALWAYS or NEVER to a rule and the agent still misses it.
• The prompt has if-then branches for specific customers, tools or workflows.
• A mistake you already fixed keeps coming back.

To see the real size, count the prompt as your production code assembles it, broken down by section. Templates and injected blocks rarely show up in the file you edit.

What should you move out of the system prompt?

Sort each line by what kind of instruction it is, then move it to the place that fits.

If the line is…	Move it to	Why
A rule that must always hold, like a refund limit or an approval	Code: validation, permissions or a tool guard	A model can decide a guideline doesn’t apply. Code runs every time.
Guidance on using one tool	That tool’s description and schema	Descriptions are loaded into the agent’s context, and Anthropic found that “Even small refinements to tool descriptions can yield dramatic improvements.”
Reference material, like policies or product details	Retrieval or a lookup tool	The agent loads it at runtime when it needs it.
An instruction for one situation, account or workflow	A conditional instruction added by your harness	It reaches only the conversations your condition matches.
True of every conversation: role, tone, core goal	The system prompt	This is what the system prompt is for.

How do you add an instruction only when it applies?

Have your harness decide. Your harness is the code that assembles the prompt and calls the model, so it can check the situation on each turn and append the matching instruction.

This is one practical piece of context engineering, which Anthropic defines as “the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference.”

Here is the idea with no framework:

BASE_PROMPT = "You help customers manage their insurance policies."

CONDITIONAL_INSTRUCTIONS = [
    {
        "applies": lambda turn: turn.workflow == "add_driver",
        "text": "Check the driver's license before you change coverage.",
    },
    {
        "applies": lambda turn: turn.account_tier == "enterprise",
        "text": "Offer the customer's account manager for billing questions.",
    },
]


def build_system_prompt(turn):
    matches = [i["text"] for i in CONDITIONAL_INSTRUCTIONS if i["applies"](turn)]
    return "\n\n".join([BASE_PROMPT, *matches])

Frameworks have their own hooks for this. LangChain has the @dynamic_prompt middleware decorator, and Pydantic AI has @agent.instructions.

Why not let the model choose?

Instructions the model picks for itself, like skills, only work if the model reaches for them. In Vercel’s evals of a coding agent using framework documentation, the skill was never invoked in 56% of cases under default behavior. Vercel concluded that always-present context works better for general framework knowledge.

That suits knowledge most runs need. For an instruction tied to one situation, a check in your own code takes the decision away from the model and keeps the instruction out of every other conversation.

Two things to watch

• Prompt caching. Put stable instructions first and additions last. OpenAI’s caching guide says cache reuse “requires the entire rendered prefix to match.” If an addition in the system prompt changes between turns, the conversation history after it misses the cache too.
• Conditions that match too much. A trigger that looks narrow can fire more often than you expect. Test it against real past conversations, including nearby ones where it shouldn’t fire.

How do you shrink a prompt safely?

1. Build a test set from real conversations. Include the failures behind the rules you plan to touch, plus ordinary cases.
2. Measure a baseline across several runs. Anthropic’s example: a task with a 75% per-trial success rate has only about a 42% chance of passing three trials in a row.
3. Move or cut one group of related lines at a time. Use the table above to decide where each goes.
4. Compare the results. Keep the change if results hold. If they don’t, restore the lines and write down why they exist.
5. Require a scope for every new line. Before adding a rule, ask which conversations need it. If the answer isn’t all of them, it doesn’t belong in the system prompt.

Where Mirrors fits

Mirrors is a developer tool for production AI agents, and it handles the situation and workflow cases in that table. It surfaces recurring mistakes across your agent’s runs. You or your coding agent write a correction, a short instruction for one situation, and preview the stored conversations it would have reached. When its trigger matches, the SDK returns the correction and your harness appends it to the next model request’s system prompt. A holdout keeps a share of matching sessions without it, so you can compare outcomes using the metrics your team tracks.

Preview shows where a correction would apply. It doesn’t prove the correction helps, so measure that separately.

If you only have a few simple conditions, the function above may be all you need. Mirrors is for teams whose list of patches has outgrown what they can track by hand. See how it works or read the corrections guide.