Error-path presets
Error-path presets fork a golden where a tool or the provider fails, and check that the agent's error path still does the right thing.
A recording of a healthy run never holds a tool timeout or a rate limit, so a golden never tests what the agent does then. General fuzzing would, but every mutation changes the next request, so each one costs a live run. And most of what fuzzing finds is a failure no real system produces.
On this page
Run the presets on your goldens
#Six presets cover the failures agents meet in production. Run them on every golden:
shadowlap test goldens/* --chaosIt prints a line per preset, and the checks each attempt was held to:
⚡ goldens/forecast: 6 error-path presets at get_forecast (#3) and model call #4
✓ tool timeout passed timed out: the agent exited 0, 1 check passed
✓ empty result passed answered with nothing: the agent exited 0, 1 check passed
✗ malformed JSON failed answered malformed JSON: the agent exited with code 1
SyntaxError: Unterminated string in JSON at position 52 (line 1 column 53)
✓ oversized result passed answered 1,000,000 characters: the agent exited 0, 1 check passed
✓ provider 429 passed failed with HTTP 429: the agent exited 0, 1 check passed
✓ provider 500 passed failed with HTTP 500: the agent exited 0, 1 check passed
checks: send_alert called at most 1 time (the write limits of the golden's expect)
5 passed, 1 failed · 2 upstream requestsThe presets
#Each preset plays one failure in:
| Preset | What the agent gets |
|---|---|
timeout | The tool times out: it doesn't run, and the agent's SDK raises the timeout its tool would (TimeoutError, or an HTTP library's own) |
empty | The tool answers with nothing: an empty list, object or text, whichever it recorded |
malformed | The tool answers with its JSON cut short, as a response that stopped arriving is |
oversized | The tool answers with its longest text repeated until the answer is a million characters long |
429 | The model call that reads the tool's answer is rate-limited, every attempt of it, so the SDK's retries run out |
500 | The same, with a server error |
Three of them in more detail:
empty: an HTTP tool gets an empty body of its type, and an MCP tool no content.malformed: a tool that returned data gets the cut JSON text in its place.oversized: the answer is far past what a model's context takes. JSON stays JSON: its longest string grows, or its longest list.
Where they hit
#Every preset hits one place in the recording, so each fork tells you about one failure:
| Presets | What they hit |
|---|---|
| Tool | The first tool call that reads the world: a read tool, else one whose effect nothing declares, else a write |
| Provider | The model call after it, the one that would read its answer |
- The tool presets never hit a
puretool, which is your own code. They never hit what a person typed, or a call that already failed when it was recorded. --chaos-tool <name>(orchaos.tool) makes them hit the first call of another tool.- The provider presets hit the agent mid-task, with state built up. With no tool call to hit, they hit the first model call.
Everything before that call is served from the recording, for free.
How a preset is judged
#Each preset is a fork: from the call it hits, the agent runs live. So, like a golden that forks, it needs the providers, and its later model calls spend tokens.
Writes after the fork are stubbed: the agent is told they succeeded, and nothing is sent. A golden
that forks blocks them instead, but a blocked write raises an error in the agent, and every preset
would then test that error too. Stubbed, the preset is the only failure played in. --writes block
blocks them anyway.
The checks an attempt must pass
#An attempt passes when the agent exits 0 and passes the checks an error path must pass. Not all of
the golden's own expect carries over. It describes the path its recording took (a refund issued,
an email sent), and an agent that escalates when the order lookup times out is right not to take it.
By default, the checks are the golden's write limits:
- Upper bounds hold. The upper bounds the golden's
expectsets on write tools (and tools whose effect nothing declares) hold on any path:"issue_refund": 1becomes "at most once", and"send_email": 0stays "never". - Lower bounds, and bounds on reads, are left out. They describe the recorded path: a read retried after a timeout is the agent doing its job.
With chaos.expect, its checks apply instead, in the grammar of expect:
{
"tests": {
"goldens/refund-in-limit": {
"expect": { "tools": { "issue_refund": 1, "send_email": 1 } },
"chaos": { "expect": { "tools": { "issue_refund": "<= 1", "escalate": "<= 1" } } }
}
}
}The last line of the printed table names the checks and where they came from. A golden whose
expect sets no upper bound on a write, and that has no chaos.expect, is judged on the agent
exiting 0 alone, and the command says so.
Verdicts
#A preset's verdict is one of four:
| Verdict | When |
|---|---|
| passed | The gate below passes it |
| failed | The gate below fails it |
| inconclusive | The gate ran out of attempts, the call was never reached, a check couldn't be decided, or the provider stayed unavailable |
| not applicable | The recording has no call the preset can hit, or no answer it can change |
An inconclusive preset is one of these:
- The gate ran out of attempts between its bounds.
- The replay never reached the call: the golden no longer replays as far as that, and
shadowlap testsays why. - A check could not be decided, such as a judge with no way to be asked, which no further attempt would decide either.
- The provider stayed unavailable too many attempts in a row, counted across the session's presets. That leaves every preset after it inconclusive too.
An answer a preset can't change is a binary response, a result with no text or list to grow, or a tool that already answered with nothing.
A preset whose change never reaches a model request replays clean from there: it was played in, and the agent coped. That happens when the agent reads one field of the answer, and that field is still there.
Gating
#Forks vary, so a preset can be decided over several attempts, as they come in:
- By default, by the golden's
repeatandpasses, as its own forks are. - With
chaos.gate, by a sequential test, below.
A sequential test
#{ "good": 0.9, "bad": 0.5, "risk": 0.1, "max": 6 } tells an agent that copes 90% of the time or
more from one that copes 50% or less. Each verdict is wrong at most 10% of the time at those two
rates; between them, either can come.
The output states the gate in two places:
- Each preset's line shows its tally and the bound of its pass rate, exact under the gate's
stopping:
timed out: 4 of 4 attempts passed · rate >= 0.47(90%). - The printed table states the rule once under it:
gate: tells 90%+ from 50%- at <= 10% risk, up to 6attempts.
Each passing attempt adds ln(good/bad) to its score, and each failure ln((1−good)/(1−bad)). The
preset passes once the score reaches ln(1/risk), and fails once it falls to ln(risk): with these
figures, after four passes or two failures. It is inconclusive after max attempts.
Every figure is optional, as for a golden's own gate:
| Figure | Default |
|---|---|
good | 0.9 |
bad | 0.7 |
risk | 0.05 |
max | 80 |
A max too small for an agent that always passes to reach the pass bound is refused, since the gate
could never pass.
Inconclusive presets
#--inconclusive fail(the default) fails the command on an inconclusive preset.--inconclusive warnreports it, marks its JUnit case skipped, and lets it through, so the exit status counts it as passed.
Configure a golden's presets
#Set these under chaos in the golden's tests entry in shadowlap.config.json. Every key is
optional:
| Key | What it says | Default |
|---|---|---|
presets | Which presets test --chaos runs on it, such as ["timeout", "429"]; --presets overrides it | All six |
expect | What every attempt must pass besides exiting 0, in the grammar of expect | The write limits of the golden's own expect |
tool | The tool whose first call the tool presets hit | The first that reads the world |
gate | { good, bad, risk, max }, the sequential test | The golden's repeat and passes |
oversize | How many characters an oversized answer grows to | 1000000 |
timeout | Seconds an attempt may run before it is stopped, and fails: an error path can hang | 300 |
"chaos": false leaves the golden out of test --chaos. Asked for by name, replay --chaos on one
of its runs still runs the presets, with the defaults.
On any run
#Run the presets on any run of the store with replay --chaos:
shadowlap replay <run> --chaos --expect '{"tools":{"issue_refund":"<= 1"}}'It judges them by --expect, or, for a run of a golden, by that golden's chaos.expect or write
limits. It doesn't combine with the flags that change a replay, such as --override or --fail-at:
each preset is played in alone.
In CI
#Presets cost tokens, so they suit a nightly job better than every pull request:
shadowlap test goldens/* --chaos --junit chaos.xml --markdown "$GITHUB_STEP_SUMMARY"Each report flag writes the presets' results:
| Flag | What it writes |
|---|---|
--junit | A case per preset |
--markdown | A table of them, with any golden whose presets couldn't run listed under it |
--json | The report, with each attempt's run and verdict (its fields) |
A JUnit case that fails lists each attempt by its own verdict. An attempt whose replay never reached the call, or that only an undecided check held back, reads inconclusive there too.
Under a chaos.gate, the Markdown table states its rule once under the rows. Each JUnit case the
gate decided states it after its tally and bound.
What is kept
#The store keeps each session and every preset's result, per golden: the call it hit, the verdict, the attempts and their runs.
Each attempt is a run of its own, bundled under the recording it forked, as blame's probes are. So
Runs lists the recording, and shadowlap show on an attempt opens the error path the agent took.
The store also names the process running each session until the session ends, as it does for a test session. So the dashboard reads presets started in a terminal as running. While they run on goldens, the jobs that replay or rewrite the goldens wait for them and name the command: Run tests, Update golden and Find breaking commit. A session whose process was killed holds nothing up: it reads as one that did not finish.
Limits
#- One place per preset. The presets hit the first read and the model call after it. A failure
later in the run is a fork of its own:
--fail-atand--overrideplay one in where you choose. - An oversized answer costs what it costs. Most models refuse a request past their context, at
no charge; one with a context of a million tokens reads it. Lower
chaos.oversize, or leave the preset out, for those. - Every attempt of the failed model call fails. The SDK's retries meet the same failure, and so
does the agent's own. One that retries for ever runs until the attempt's time limit (
timeout, 5 minutes by default), and fails.
See also
#- CI: goldens that fork on purpose.
- Gating: sequential tests and their verdicts.
- Checks: the grammar of
expect. - Interventions: one failure, played in where you choose.