Contents
More

Error-path presets

Error-path presets fork a golden where a tool or the provider fails, and check that the agent's error path still does the right thing.

A recording of a healthy run never holds a tool timeout or a rate limit, so a golden never tests what the agent does then. General fuzzing would, but every mutation changes the next request, so each one costs a live run. And most of what fuzzing finds is a failure no real system produces.

On this page

Run the presets on your goldens

#

Six presets cover the failures agents meet in production. Run them on every golden:

shadowlap test goldens/* --chaos

It prints a line per preset, and the checks each attempt was held to:

Text
⚡ goldens/forecast: 6 error-path presets at get_forecast (#3) and model call #4
  ✓ tool timeout      passed  timed out: the agent exited 0, 1 check passed
  ✓ empty result      passed  answered with nothing: the agent exited 0, 1 check passed
  ✗ malformed JSON    failed  answered malformed JSON: the agent exited with code 1
      SyntaxError: Unterminated string in JSON at position 52 (line 1 column 53)
  ✓ oversized result  passed  answered 1,000,000 characters: the agent exited 0, 1 check passed
  ✓ provider 429      passed  failed with HTTP 429: the agent exited 0, 1 check passed
  ✓ provider 500      passed  failed with HTTP 500: the agent exited 0, 1 check passed
  checks: send_alert called at most 1 time (the write limits of the golden's expect)

5 passed, 1 failed · 2 upstream requests

The presets

#

Each preset plays one failure in:

PresetWhat the agent gets
timeoutThe tool times out: it doesn't run, and the agent's SDK raises the timeout its tool would (TimeoutError, or an HTTP library's own)
emptyThe tool answers with nothing: an empty list, object or text, whichever it recorded
malformedThe tool answers with its JSON cut short, as a response that stopped arriving is
oversizedThe tool answers with its longest text repeated until the answer is a million characters long
429The model call that reads the tool's answer is rate-limited, every attempt of it, so the SDK's retries run out
500The same, with a server error

Three of them in more detail:

  • empty: an HTTP tool gets an empty body of its type, and an MCP tool no content.
  • malformed: a tool that returned data gets the cut JSON text in its place.
  • oversized: the answer is far past what a model's context takes. JSON stays JSON: its longest string grows, or its longest list.

Where they hit

#

Every preset hits one place in the recording, so each fork tells you about one failure:

PresetsWhat they hit
ToolThe first tool call that reads the world: a read tool, else one whose effect nothing declares, else a write
ProviderThe model call after it, the one that would read its answer
  • The tool presets never hit a pure tool, which is your own code. They never hit what a person typed, or a call that already failed when it was recorded.
  • --chaos-tool <name> (or chaos.tool) makes them hit the first call of another tool.
  • The provider presets hit the agent mid-task, with state built up. With no tool call to hit, they hit the first model call.

Everything before that call is served from the recording, for free.

How a preset is judged

#

Each preset is a fork: from the call it hits, the agent runs live. So, like a golden that forks, it needs the providers, and its later model calls spend tokens.

Writes after the fork are stubbed: the agent is told they succeeded, and nothing is sent. A golden that forks blocks them instead, but a blocked write raises an error in the agent, and every preset would then test that error too. Stubbed, the preset is the only failure played in. --writes block blocks them anyway.

The checks an attempt must pass

#

An attempt passes when the agent exits 0 and passes the checks an error path must pass. Not all of the golden's own expect carries over. It describes the path its recording took (a refund issued, an email sent), and an agent that escalates when the order lookup times out is right not to take it.

By default, the checks are the golden's write limits:

  • Upper bounds hold. The upper bounds the golden's expect sets on write tools (and tools whose effect nothing declares) hold on any path: "issue_refund": 1 becomes "at most once", and "send_email": 0 stays "never".
  • Lower bounds, and bounds on reads, are left out. They describe the recorded path: a read retried after a timeout is the agent doing its job.

With chaos.expect, its checks apply instead, in the grammar of expect:

JSON
{
  "tests": {
    "goldens/refund-in-limit": {
      "expect": { "tools": { "issue_refund": 1, "send_email": 1 } },
      "chaos": { "expect": { "tools": { "issue_refund": "<= 1", "escalate": "<= 1" } } }
    }
  }
}

The last line of the printed table names the checks and where they came from. A golden whose expect sets no upper bound on a write, and that has no chaos.expect, is judged on the agent exiting 0 alone, and the command says so.

Verdicts

#

A preset's verdict is one of four:

VerdictWhen
passedThe gate below passes it
failedThe gate below fails it
inconclusiveThe gate ran out of attempts, the call was never reached, a check couldn't be decided, or the provider stayed unavailable
not applicableThe recording has no call the preset can hit, or no answer it can change

An inconclusive preset is one of these:

  • The gate ran out of attempts between its bounds.
  • The replay never reached the call: the golden no longer replays as far as that, and shadowlap test says why.
  • A check could not be decided, such as a judge with no way to be asked, which no further attempt would decide either.
  • The provider stayed unavailable too many attempts in a row, counted across the session's presets. That leaves every preset after it inconclusive too.

An answer a preset can't change is a binary response, a result with no text or list to grow, or a tool that already answered with nothing.

A preset whose change never reaches a model request replays clean from there: it was played in, and the agent coped. That happens when the agent reads one field of the answer, and that field is still there.

Gating

#

Forks vary, so a preset can be decided over several attempts, as they come in:

  • By default, by the golden's repeat and passes, as its own forks are.
  • With chaos.gate, by a sequential test, below.

A sequential test

#

{ "good": 0.9, "bad": 0.5, "risk": 0.1, "max": 6 } tells an agent that copes 90% of the time or more from one that copes 50% or less. Each verdict is wrong at most 10% of the time at those two rates; between them, either can come.

The output states the gate in two places:

  • Each preset's line shows its tally and the bound of its pass rate, exact under the gate's stopping: timed out: 4 of 4 attempts passed · rate >= 0.47 (90%).
  • The printed table states the rule once under it: gate: tells 90%+ from 50%- at <= 10% risk, up to 6 attempts.

Each passing attempt adds ln(good/bad) to its score, and each failure ln((1−good)/(1−bad)). The preset passes once the score reaches ln(1/risk), and fails once it falls to ln(risk): with these figures, after four passes or two failures. It is inconclusive after max attempts.

Every figure is optional, as for a golden's own gate:

FigureDefault
good0.9
bad0.7
risk0.05
max80

A max too small for an agent that always passes to reach the pass bound is refused, since the gate could never pass.

Inconclusive presets

#
  • --inconclusive fail (the default) fails the command on an inconclusive preset.
  • --inconclusive warn reports it, marks its JUnit case skipped, and lets it through, so the exit status counts it as passed.

Configure a golden's presets

#

Set these under chaos in the golden's tests entry in shadowlap.config.json. Every key is optional:

KeyWhat it saysDefault
presetsWhich presets test --chaos runs on it, such as ["timeout", "429"]; --presets overrides itAll six
expectWhat every attempt must pass besides exiting 0, in the grammar of expectThe write limits of the golden's own expect
toolThe tool whose first call the tool presets hitThe first that reads the world
gate{ good, bad, risk, max }, the sequential testThe golden's repeat and passes
oversizeHow many characters an oversized answer grows to1000000
timeoutSeconds an attempt may run before it is stopped, and fails: an error path can hang300

"chaos": false leaves the golden out of test --chaos. Asked for by name, replay --chaos on one of its runs still runs the presets, with the defaults.

On any run

#

Run the presets on any run of the store with replay --chaos:

shadowlap replay <run> --chaos --expect '{"tools":{"issue_refund":"<= 1"}}'

It judges them by --expect, or, for a run of a golden, by that golden's chaos.expect or write limits. It doesn't combine with the flags that change a replay, such as --override or --fail-at: each preset is played in alone.

In CI

#

Presets cost tokens, so they suit a nightly job better than every pull request:

shadowlap test goldens/* --chaos --junit chaos.xml --markdown "$GITHUB_STEP_SUMMARY"

Each report flag writes the presets' results:

FlagWhat it writes
--junitA case per preset
--markdownA table of them, with any golden whose presets couldn't run listed under it
--jsonThe report, with each attempt's run and verdict (its fields)

A JUnit case that fails lists each attempt by its own verdict. An attempt whose replay never reached the call, or that only an undecided check held back, reads inconclusive there too.

Under a chaos.gate, the Markdown table states its rule once under the rows. Each JUnit case the gate decided states it after its tally and bound.

What is kept

#

The store keeps each session and every preset's result, per golden: the call it hit, the verdict, the attempts and their runs.

Each attempt is a run of its own, bundled under the recording it forked, as blame's probes are. So Runs lists the recording, and shadowlap show on an attempt opens the error path the agent took.

The store also names the process running each session until the session ends, as it does for a test session. So the dashboard reads presets started in a terminal as running. While they run on goldens, the jobs that replay or rewrite the goldens wait for them and name the command: Run tests, Update golden and Find breaking commit. A session whose process was killed holds nothing up: it reads as one that did not finish.

Limits

#
  • One place per preset. The presets hit the first read and the model call after it. A failure later in the run is a fork of its own: --fail-at and --override play one in where you choose.
  • An oversized answer costs what it costs. Most models refuse a request past their context, at no charge; one with a context of a million tokens reads it. Lower chaos.oversize, or leave the preset out, for those.
  • Every attempt of the failed model call fails. The SDK's retries meet the same failure, and so does the agent's own. One that retries for ever runs until the attempt's time limit (timeout, 5 minutes by default), and fails.

See also

#
  • CI: goldens that fork on purpose.
  • Gating: sequential tests and their verdicts.
  • Checks: the grammar of expect.
  • Interventions: one failure, played in where you choose.

This page on GitHub →