change safety for agents

Race your
AI agent against
its last run.

Every run leaves a shadow. Replay a failure exactly, find the call that lost the run, and prove your fix holds before it ships.

$npm install -g shadowlap
Open source · TypeScript and Python · no account needed
works with The stack you already run
  • providers
  • Anthropic
  • OpenAI
  • Gemini
  • Agent Platform
  • Amazon Bedrock
  • Azure OpenAI
  • Claude on Microsoft Foundry
  • Ollama
  • Vercel AI Gateway
  • OpenRouter
  • frameworks
  • Vercel AI SDK
  • OpenAI Agents SDK
  • Mastra
  • LangChain
  • LangGraph
  • Anthropic tool runner
  • Claude Agent SDK
  • Pydantic AI
  • CrewAI
  • Google ADK
  • Microsoft Agent Framework
  • LlamaIndex
  • Strands Agents
  • smolagents
  • Agno
  • coding agents
  • Claude Code
  • Cursor
  • Codex
  • GitHub Copilot
  • Antigravity
  • Devin Desktop
  • imports from
  • OpenTelemetry
  • Arize Phoenix
  • Langfuse
  • LangSmith
  • Logfire
  • LiteLLM

Your agent refunded $240 it should have escalated.

Run it again and it behaves, so you can't prove a fix worked.

replay · how it works

One lap to record.
Every lap after that runs the same line.

Put shadowlap in front of the agent you already run. Every replay comes out the same, so the first call that differs is the one your change touched.

lap 3 of 3 live from call 17 · writes fenced call 30/30
~ $.messages[14].content[0].content
+ "{…\"refund_limit\":100,…}"

One track, three passes over it. The run is recorded live, once: 30 calls. It is replayed exactly, sending nothing upstream. Then call 17 changes: the 16 calls before it replay, and from call 17 the run goes on live, with writes fenced.

  1. 01 · record

    Record at the HTTP layer.

    A local sidecar sits in front of the model API, and shadowlap run points your SDK at it: no code changes, and keys are never stored.

    $shadowlap run -- node refund_agent.mjs
  2. 02 · replay

    Replay it exactly, in any order.

    Calls match by fingerprint, not arrival order, and the clock and random values the SDK reads replay too. An unchanged run sends nothing upstream.

    $shadowlap replay 01K5CQ4N7G
  3. 03 · fork

    Fork at the first change.

    The first request that differs gets a diff. From there the run goes on live, with writes fenced, so a fork won’t issue a real refund.

    $shadowlap diff 01K5CQ4N7G 01K5CQ4P2R

Re-run live, the run’s 14 model requests would send 20.3k input tokens. The replay sends none.

integrations · import · wrap

Start from traces,
or from your code.

Model calls need no code changes. Import the traces you already keep, or wrap each tool once, so a fork never issues a real refund.

terminal · ~/support-agent
$shadowlap import otel genai-traces.jsonImported 01M5J43QM2 from genai-traces.json: 2 model calls and 1 tool result (2 anthropic).Read 5 spans. It kept the messages of 2 of 2 model calls, settings for 2, tool definitions for 2, and the output of 1 of 1 tool call.A trace says what was sent, not what sent it. Give your own command:
then$shadowlap replay 01M5J43QM2 -- node agent.mjs

Keys are read from your environment, never stored. Langfuse, Arize Phoenix and Logfire import a window of traces at once, and OpenLLMetry, HAR, VCR and nock import too. Every import →

fixes · blame · repair · verify

Find the call that lost the run.
Ship a fix that holds.

Not sure which call to change? shadowlap finds it, drafts a fix, and keeps it only if the good runs still pass. Repeated runs decide each verdict, at a stated risk.

race controlfix holds · 3 of 3
  • off at turn 5 · issue_refund({…"amount":240})
  • fixed run · takes the apex · held
off at turn 5 · lost
issue_refund({…"amount":240})
#5 · blamed · rescued 5 of 10 new line fixed run · held
blame · asked again
#1 rescued 5/10
#5 5/10 · blamed
#7 0/6 · lost
checked#5 11/22 · #7 0/22
repair · the new line

…Follow the refund policy.

…Follow the refund policy: escalate every refund above the limit.

verify · 5% risk each
target12/12
held-out17/18
good38/38
baseline 38/38
accepted · +1.9% cost/run
  1. 01 · blame

    Which call lost the run?

    Blame replays free up to a model call, asks it again, and names the last one that can still save the run.

    $shadowlap blame 01M3RZ4XKN --expect @refund-check.json
  2. 02 · repair

    What should the prompt say?

    A model you choose, or your Claude Code, rewrites the prompt. The smallest rewrite that passes wins.

    $shadowlap repair --fixes label:refund-bug --keeps label:good --expect @refund-check.json --repairer claude-code
  3. 03 · verify

    Does the fix hold everywhere?

    Failing and unseen runs must be fixed; good runs must not get worse. The fix comes with its proof.

    $shadowlap verify --fixes 01M3RXQ4VB --holdout 01M3RXQ51B --keeps label:good --expect @refund-check.json
ci · shadowlap test

Every PR races the last good run.

CI replays the runs you commit, with no key until a fork, and fails the check when a request changes, a rule breaks or a pass rate drops.

  • expect

    Rules on what it did.

    Order, conditions and values from earlier calls, as well as tools and cost.

  • judge

    Quality, judged once.

    A model judges the answer; its verdict is recorded once and replays free.

  • gate

    Rates, not counts.

    Forked goldens pass on a rate, with one false-alarm risk per suite.

  • comment

    The verdict on the PR.

    Each golden's verdict in one comment, updated on every push.

  • --update

    Goldens that keep up.

    One flag re-records a golden after a change you meant to make.

  • fix-on-failure

    A fix, if you want one.

    When a rule breaks, CI can open a pull request with a verified fix.

  • migrate --to

    New model? Race it.

    It answers every recorded question; only a write, stop or judged change is verified live.

  • decisions --from-code

    Edits, previewed.

    Which recorded decisions a prompt or tool edit changes, before CI replays.

for coding agents

Hand the wheel to your coding agent.

Claude Code, Codex, Cursor and Copilot set shadowlap up for your agent, and check each change they make before calling it done.

claude code · ~/support-agent
The refund agent paid out $240 it should have escalated. Find out why and fix it.
shadowlap ls --label refund-bug --json
"runId": "01M3RZ4XKN…" … "labels": ["refund-bug"]
shadowlap blame 01M3RZ4XKN --expect @refund-check.json --json
"confidence": "high" … "callNo": 5
shadowlap repair --fixes label:refund-bug --keeps label:good --expect @refund-check.json --repairer claude-code
✓ Fixed: Escalate refunds above the policy limit
The system prompt never said what the policy asks of large refunds, so at call #5 the model reached for issue_refund. The fix adds the escalation rule: it fixes the two failing runs and the one held out, and the good run still passes.
  • Plain CLI: every command they need reads and writes JSON
  • Fix what lost the run: blame, repair, verify
  • Writes stay fenced after every fork, whoever is driving
Paste once into CLAUDE.md or AGENTS.md, or run shadowlap skill install
shadowlap dashboard

Step through any run,
call by call.

Open a run on localhost: what the model saw at each call, where the replay forked, and what it cost.

Also in the dashboard
  • ContextTokens per call by cache tier, and every call where the cache stopped hitting.
  • CompareAny two runs, from what changed to the first request that differs.
  • SearchPrompts, responses, tool arguments and results, across every run.
terminal ui · shadowlap ui

Or never leave the terminal.

The same runs, in your terminal. Step to the call that lost one and fork it. Verify the fix without touching the mouse.

also in the box

Everything else a recording is good for.

A recording is also a test, a dataset and an incident you can step through.

  • shadowlap import

    A production incident, replayed.

    Capture from your gateway or import a trace you already have, then replay your own code against it.

  • --inject · --override · --model · --fail-at

    Ask “what if?”

    Change a tool result, the model or an error at step k. The steps before it replay free.

  • shadowlap push · pull

    Send a teammate the run.

    Push runs to your own bucket. A teammate pulls one and replays it exactly.

  • @pytest.mark.shadowlap

    Agent tests that don't flake.

    Tests record once, then replay offline: identical every time, fast and free.

  • shadowlap bisect

    Which commit broke it?

    Bisect replays a golden offline, halving the range, and names the first commit that breaks.

  • shadowlap export pairs

    Every fixed failure is a training pair.

    Each fix leaves a losing answer and a passing one for the same conversation, ready to export.

  • shadowlap injection

    Did planted text cause it?

    Plant a canary in an untrusted read and replay. A tool call that carries it is proof.

  • --chaos

    Rehearse the errors.

    Replay a golden through six common failures and see how your agent handles each.

  • shadowlap coverage

    What no golden covers.

    Production runs, grouped by what they did. Turn any behaviour no golden covers into one with --pack.

A trace says what happened. A fork says what would have happened.

compare · where it fits

Exact until
the call that changed.
Live from there.

Keep your tracing and evals. Add the proof. Runs export to Langfuse and Braintrust.

Capability shadowlapreplay, fork, verify Tracing and evalsspans, logs, dashboards Replay evalsscored experiments
The question it answers
Why did it fail? Is the fix real?
What happened in production?
Is a new model better at scale?
Finds the call that lost a run
YesBlame re-asks calls from forks and names the last one that could still have saved the run, with a confidence.
PartlyLangSmith's Polly and Engine, Braintrust's Loop and Phoenix's PXI read traces and propose a cause.
PartlyA failure matrix shows where failed sessions first go wrong: at an error, or at a step a reviewer marks.
Proves a fix holds
YesVerify accepts a change only when the failing runs pass and the good ones still do. Repair asks a model, or your own Claude Code, to rewrite the prompt or a tool description.
PartlyLangSmith Engine replays up to five traces live against the pull request it opens, for agents deployed on LangSmith Cloud, in a private beta.
PartlyAn experiment scores your change across a cohort of sessions, with the model live.
Forks at the first change
YesEvery call before the first change replays, and that request gets a structural diff. From there the run goes on live.
No
NoEvery replay starts again from the top of the session.
tracing and evals

Watch production, and score a new model across hundreds of sessions.

shadowlap

Find the call that lost a run, prove the fix, gate the PR.

Seven more questions, and cassettes, agent debuggers, framework time travel and durable execution, are in the docs. The full comparison →

open source · local-first

Recorded on your machine.
Nothing held back.

Every part is in the repository, under Apache-2.0. No account, no tracking, no feature gate.

$npm install -g shadowlap
Source on GitHub
  • credentials

    Keys pass through.

    They go to the provider and are never stored. A clean replay needs none.

  • 127.0.0.1

    Local only.

    The proxy and the dashboard listen on this machine, behind a session token.

  • .shadowlap/

    Yours to read.

    Recordings stay in your project, readable only by you and kept out of git.

  • redact

    Masked on the way out.

    Your rules run on every pack, export and share link. A push copies runs as recorded, keys masked.

  • unpack

    Shared runs are checked.

    Every file is verified before anything is written, and replaying one asks before running its command.

  • telemetry

    Off by default.

    Nothing is sent unless you point it at a collector you run.

A run leaves your machine only when you export, share or push it, or ask a model about it. Report a vulnerability →

faq

What teams ask first.

Product
How is this different from LangSmith, Langfuse or Braintrust?

They record what happened so you can read and score it. shadowlap re-runs your real code against a recording, finds the call that lost the run and proves the fix. A Langfuse or LangSmith trace imports as a recording, and runs export to Langfuse and Braintrust.

And from replay-based evals like Kitaru?

They ask the model again on every replay, so every replay spends tokens and starts from the top. shadowlap replays the model too: free until the call that changed, which forks with a diff.

And from agent debuggers like Laminar or OrcaReplay?

They replay the model's recorded answers too, up to a call you choose, but run the tools again. shadowlap replays the tools as well, forks at the first request your change alters, fences writes, and adds blame, verify and offline CI.

The model isn't deterministic. How can a replay be exact?

The model isn't asked again: its answers come from the recording, byte for byte. After a fork, verify replays each run until a gate can tell a real fix from a lucky run.

How sure is a verified fix?

Verify replays failing and unseen runs until it tells a 90% pass rate from 70%, and good runs may not drop 10 points. Each verdict is wrong at most 5% of the time, and every figure is yours to set.

What does finding and proving a fix cost?

Only the calls after a fork run live: everything before it replays free, and a run your change doesn't touch costs nothing. --budget limits what blame, repair and verify spend, and each report says what it spent.

My agent sends emails and issues refunds. Is replaying it safe?

Yes, for the tools you wrap: after a fork, write tools are blocked, or stubbed with a value you choose. A replay won't start if the run called an unwrapped tool that could write.

How do you test prompt injection?

shadowlap plants a canary in an untrusted read, in a plain instruction or inside your red-teaming tool's payloads, and replays with writes fenced. A tool call that carries the canary proves the planted text caused it. It complements red teaming; it doesn't replace it.

We have to switch models by a retirement date. What changes?

Run shadowlap migrate on your goldens before the date. It shows each decision the new model makes differently, changed writes first, and checks live only the runs that write, stop or are judged differently. The report ends with what still passes, what it costs and what to set.

Does it work with chat agents that wait for a user?

Yes. What the user types is recorded like a tool's result, so a conversation replays, tests and blames without anyone typing.

Do I have to change my agent's code?

Not for model calls: shadowlap run points your SDK at its proxy. Each tool is wrapped once, in a line or by your framework's adapter, and shadowlap init finds them. MCP servers need none.

Do you train on our runs?

No. Nothing is collected by default and runs are never pooled; they stay in your project. Whether you train on your recordings, or on pairs you export, is between you and your model provider.

Open source
Will everything stay free?

Yes. Everything in shadowlap, the team store in your own bucket included, is free under Apache-2.0, and what is free stays free.

Every bad run becomes the lap your fix has to beat.

Reproduce it exactly. Prove the fix. Keep it fixed.

Get started →
$npm install -g shadowlap
Open source · Apache-2.0 · Source on GitHub →