Race your
AI agent against
its last run.
Every run leaves a shadow. Replay a failure exactly, find the call that lost the run, and prove your fix holds before it ships.
$npm install -g shadowlap- providers
- Anthropic
- OpenAI
- Gemini
- Agent Platform
- Amazon Bedrock
- Azure OpenAI
- Claude on Microsoft Foundry
- Ollama
- Vercel AI Gateway
- OpenRouter
- frameworks
- Vercel AI SDK
- OpenAI Agents SDK
- Mastra
- LangChain
- LangGraph
- Anthropic tool runner
- Claude Agent SDK
- Pydantic AI
- CrewAI
- Google ADK
- Microsoft Agent Framework
- LlamaIndex
- Strands Agents
- smolagents
- Agno
- coding agents
- Claude Code
- Cursor
- Codex
- GitHub Copilot
- Antigravity
- Devin Desktop
- imports from
- OpenTelemetry
- Arize Phoenix
- Langfuse
- LangSmith
- Logfire
- LiteLLM
Your agent refunded $240 it should have escalated.
Run it again and it behaves, so you can't prove a fix worked.
One lap to record.
Every lap after that runs the same line.
Put shadowlap in front of the agent you already run. Every replay comes out the same, so the first call that differs is the one your change touched.
One track, three passes over it. The run is recorded live, once: 30 calls. It is replayed exactly, sending nothing upstream. Then call 17 changes: the 16 calls before it replay, and from call 17 the run goes on live, with writes fenced.
-
01 · record
Record at the HTTP layer.
A local sidecar sits in front of the model API, and
shadowlap runpoints your SDK at it: no code changes, and keys are never stored.$shadowlap run -- node refund_agent.mjs -
02 · replay
Replay it exactly, in any order.
Calls match by fingerprint, not arrival order, and the clock and random values the SDK reads replay too. An unchanged run sends nothing upstream.
$shadowlap replay 01K5CQ4N7G -
03 · fork
Fork at the first change.
The first request that differs gets a diff. From there the run goes on live, with writes fenced, so a fork won’t issue a real refund.
$shadowlap diff 01K5CQ4N7G 01K5CQ4P2R
Re-run live, the run’s 14 model requests would send 20.3k input tokens. The replay sends none.
Start from traces,
or from your code.
Model calls need no code changes. Import the traces you already keep, or wrap each tool once, so a fork never issues a real refund.
$shadowlap import otel genai-traces.jsonImported 01M5J43QM2 from genai-traces.json: 2 model calls and 1 tool result (2 anthropic).Read 5 spans. It kept the messages of 2 of 2 model calls, settings for 2, tool definitions for 2, and the output of 1 of 1 tool call.A trace says what was sent, not what sent it. Give your own command:
$shadowlap replay 01M5J43QM2 -- node agent.mjs$shadowlap import phoenix 4a2db41deabc4aec332fb9fbfa0fc3e2 --project support-agentImported 01M5J6XAJB from phoenix:4a2db41d: 3 model calls and 2 tool results (3 anthropic).Read 6 spans. It kept the messages of 3 of 3 model calls, settings for 3, tool definitions for 3, and the output of 2 of 2 tool calls.A trace says what was sent, not what sent it. Give your own command:
$shadowlap replay 01M5J6XAJB -- python agent.py$shadowlap import langfuse 0dfb6fdc-f0b6-74dc-85b3-9c2bb5bbfde3Imported 01M5J2Y43D from langfuse:0dfb6fdc: 2 model calls (2 anthropic).Read 2 observations. It kept the messages of 2 of 2 model calls, settings for 2, tool definitions for 2.The keys were read from the environment and sent to cloud.langfuse.com only; nothing of them is stored.A trace says what was sent, not what sent it. Give your own command:
$shadowlap replay 01M5J2Y43D -- node agent.mjs$shadowlap import langsmith 3eb7e3b4-723a-7a06-8f26-12a4f6c9b6e9Imported 01M5J2Y4DY from langsmith:3eb7e3b4: 3 model calls (3 anthropic).Read 4 runs. It kept the messages of 3 of 3 model calls, settings for 3, tool definitions for 3.The keys were read from the environment and sent to api.smith.langchain.com only; nothing of them is stored.A trace says what was sent, not what sent it. Give your own command:
$shadowlap replay 01M5J2Y4DY -- node agent.mjs$shadowlap import logfire 01a16460e2f311342463057f62b79fa9Imported 01M5J6XAQ2 from logfire:01a16460: 3 model calls and 2 tool results (3 anthropic).Read 6 spans. It kept the messages of 3 of 3 model calls, settings for 3, tool definitions for 3, and the output of 2 of 2 tool calls.LOGFIRE_READ_TOKEN was read from the environment and sent to logfire-us.pydantic.dev only; nothing of it is stored.A trace says what was sent, not what sent it. Give your own command:
$shadowlap replay 01M5J6XAQ2 -- python agent.py$shadowlap import litellm litellm/*.jsonlImported 1 run, 14 model calls: 01M5J2VGXY 14 calls incident-42Streamed responses replay as one event: LiteLLM hands a callback an assembled response.A capture says what was sent, not what sent it. Give your own command:
$shadowlap replay 01M5J2VGXY -- node agent.mjsKeys are read from your environment, never stored. Langfuse, Arize Phoenix and Logfire import a window of traces at once, and OpenLLMetry, HAR, VCR and nock import too. Every import →
$npm install @shadowlap/sdkimport { tool } from '@shadowlap/sdk'; export const lookupOrder = tool('lookup_order', { effect: 'read' }, getOrder); export const issueRefund = tool('issue_refund', { effect: 'write' }, refund);$shadowlap run -- node agent.mjs$pip install shadowlapfrom shadowlap import tool # Google ADK, LlamaIndex, Microsoft Agent Framework,# Strands Agents, smolagents and Agno take these tools too@tool(effect="read")def lookup_order(order_id): ... @tool(effect="write")def issue_refund(order_id, amount): ...$shadowlap run -- python agent.py$npm install @shadowlap/adaptersimport { generateText } from 'ai';import { withEffects } from '@shadowlap/adapters/ai-sdk'; // lookupOrder, issueRefund: tool() from 'ai'const tools = withEffects( { lookup_order: lookupOrder, issue_refund: issueRefund }, { lookup_order: 'read', issue_refund: 'write' },);await generateText({ model, tools, prompt });$shadowlap run -- npx tsx agent.ts$npm install @shadowlap/adaptersimport { tool } from '@openai/agents';import { withEffect } from '@shadowlap/adapters/openai-agents'; const issueRefund = tool(withEffect({ name: 'issue_refund', description: 'Refund an order', parameters: refundArgs, execute: refund,}, 'write'));$shadowlap run -- npx tsx agent.ts$npm install @shadowlap/adaptersimport { betaZodTool } from '@anthropic-ai/sdk/helpers/beta/zod';import { withEffect } from '@shadowlap/adapters/anthropic'; const issueRefund = betaZodTool(withEffect({ name: 'issue_refund', description: 'Refund an order', inputSchema: refundArgs, run: refund,}, 'write'));$shadowlap run -- npx tsx agent.ts$npm install @shadowlap/adaptersimport { query } from '@anthropic-ai/claude-agent-sdk';import { withFence } from '@shadowlap/adapters/claude-agent-sdk' ; // Bash, Write and Edit run inside Claude Code:// the fence records them, and blocks or stubs them after a forkconst options = withFence({ cwd: project });for await (const message of query({ prompt, options })) { // …}$shadowlap run -- npx tsx agent.ts$npm install @shadowlap/adaptersimport { createTool } from '@mastra/core/tools';import { withEffect } from '@shadowlap/adapters/mastra'; export const issueRefund = createTool(withEffect({ id: 'issue_refund', description: 'Refund an order', inputSchema: refundArgs, execute: refund,}, 'write'));$shadowlap run -- node agent.mjs$npm install @shadowlap/adaptersimport { tool } from '@langchain/core/tools';import { ToolNode } from '@langchain/langgraph/prebuilt';import { withEffect } from '@shadowlap/adapters/langchain'; const issueRefund = withEffect( tool(refund, { name: 'issue_refund', schema: refundArgs }), 'write',);const tools = new ToolNode([issueRefund]);$shadowlap run -- npx tsx agent.ts$pip install shadowlapfrom pydantic_ai import Agentfrom shadowlap import tool agent = Agent(model) @agent.tool_plain@tool(effect="write")def issue_refund(order_id: str, amount: int) -> dict: return payments.refund(order_id, amount)$shadowlap run -- python agent.py$pip install shadowlapfrom crewai.tools import toolfrom shadowlap.adapters.crewai import with_effect @tool("issue_refund")def issue_refund(order_id: str, amount: int) -> str: """Refund an order.""" return payments.refund(order_id, amount) tools = [with_effect(issue_refund, "write")]$shadowlap run -- python agent.pyA service your agent calls over HTTP needs no wrapper: mark it read or write in shadowlap.config.json. An MCP server’s tools need none either: start it through shadowlap mcp. How →
Find the call that lost the run.
Ship a fix that holds.
Not sure which call to change? shadowlap finds it, drafts a fix, and keeps it only if the good runs still pass. Repeated runs decide each verdict, at a stated risk.
- off at turn 5 · issue_refund({…"amount":240})
- fixed run · takes the apex · held
issue_refund({…"amount":240}) #5 · blamed · rescued 5 of 10 new line fixed run · held
…Follow the refund policy.
…Follow the refund policy: escalate every refund above the limit.
- 01 · blame
Which call lost the run?
Blame replays free up to a model call, asks it again, and names the last one that can still save the run.
$shadowlap blame 01M3RZ4XKN --expect @refund-check.json - 02 · repair
What should the prompt say?
A model you choose, or your Claude Code, rewrites the prompt. The smallest rewrite that passes wins.
$shadowlap repair --fixes label:refund-bug --keeps label:good --expect @refund-check.json --repairer claude-code - 03 · verify
Does the fix hold everywhere?
Failing and unseen runs must be fixed; good runs must not get worse. The fix comes with its proof.
$shadowlap verify --fixes 01M3RXQ4VB --holdout 01M3RXQ51B --keeps label:good --expect @refund-check.json
Every PR races the last good run.
CI replays the runs you commit, with no key until a fork, and fails the check when a request changes, a rule breaks or a pass rate drops.
$shadowlap test goldens/* --annotations --markdown "$GITHUB_STEP_SUMMARY"
expectRules on what it did.
Order, conditions and values from earlier calls, as well as tools and cost.
judgeQuality, judged once.
A model judges the answer; its verdict is recorded once and replays free.
gateRates, not counts.
Forked goldens pass on a rate, with one false-alarm risk per suite.
commentThe verdict on the PR.
Each golden's verdict in one comment, updated on every push.
--updateGoldens that keep up.
One flag re-records a golden after a change you meant to make.
fix-on-failureA fix, if you want one.
When a rule breaks, CI can open a pull request with a verified fix.
migrate --toNew model? Race it.
It answers every recorded question; only a write, stop or judged change is verified live.
decisions --from-codeEdits, previewed.
Which recorded decisions a prompt or tool edit changes, before CI replays.
Hand the wheel to your coding agent.
Claude Code, Codex, Cursor and Copilot set shadowlap up for your agent, and check each change they make before calling it done.
issue_refund. The fix adds the escalation rule: it fixes the two failing runs and the one held out, and the good run still passes.- Plain CLI: every command they need reads and writes JSON
- Fix what lost the run:
blame,repair,verify - Writes stay fenced after every fork, whoever is driving
CLAUDE.md or AGENTS.md, or run shadowlap skill install
Step through any run,
call by call.
Open a run on localhost: what the model saw at each call, where the replay forked, and what it cost.
- ContextTokens per call by cache tier, and every call where the cache stopped hitting.
- CompareAny two runs, from what changed to the first request that differs.
- SearchPrompts, responses, tool arguments and results, across every run.
Or never leave the terminal.
The same runs, in your terminal. Step to the call that lost one and fork it. Verify the fix without touching the mouse.
Everything else a recording is good for.
A recording is also a test, a dataset and an incident you can step through.
shadowlap importA production incident, replayed.
Capture from your gateway or import a trace you already have, then replay your own code against it.
--inject · --override · --model · --fail-atAsk “what if?”
Change a tool result, the model or an error at step k. The steps before it replay free.
shadowlap push · pullSend a teammate the run.
Push runs to your own bucket. A teammate pulls one and replays it exactly.
@pytest.mark.shadowlapAgent tests that don't flake.
Tests record once, then replay offline: identical every time, fast and free.
shadowlap bisectWhich commit broke it?
Bisect replays a golden offline, halving the range, and names the first commit that breaks.
shadowlap export pairsEvery fixed failure is a training pair.
Each fix leaves a losing answer and a passing one for the same conversation, ready to export.
shadowlap injectionDid planted text cause it?
Plant a canary in an untrusted read and replay. A tool call that carries it is proof.
--chaosRehearse the errors.
Replay a golden through six common failures and see how your agent handles each.
shadowlap coverageWhat no golden covers.
Production runs, grouped by what they did. Turn any behaviour no golden covers into one with
--pack.
A trace says what happened. A fork says what would have happened.
Exact until
the call that changed.
Live from there.
Keep your tracing and evals. Add the proof. Runs export to Langfuse and Braintrust.
| Capability | shadowlapreplay, fork, verify | Tracing and evalsspans, logs, dashboards | Replay evalsscored experiments |
|---|---|---|---|
| The question it answers | Why did it fail? Is the fix real? | What happened in production? | Is a new model better at scale? |
| Finds the call that lost a run | YesBlame re-asks calls from forks and names the last one that could still have saved the run, with a confidence. | PartlyLangSmith's Polly and Engine, Braintrust's Loop and Phoenix's PXI read traces and propose a cause. | PartlyA failure matrix shows where failed sessions first go wrong: at an error, or at a step a reviewer marks. |
| Proves a fix holds | YesVerify accepts a change only when the failing runs pass and the good ones still do. Repair asks a model, or your own Claude Code, to rewrite the prompt or a tool description. | PartlyLangSmith Engine replays up to five traces live against the pull request it opens, for agents deployed on LangSmith Cloud, in a private beta. | PartlyAn experiment scores your change across a cohort of sessions, with the model live. |
| Forks at the first change | YesEvery call before the first change replays, and that request gets a structural diff. From there the run goes on live. | No | NoEvery replay starts again from the top of the session. |
Watch production, and score a new model across hundreds of sessions.
Find the call that lost a run, prove the fix, gate the PR.
Seven more questions, and cassettes, agent debuggers, framework time travel and durable execution, are in the docs. The full comparison →
Recorded on your machine.
Nothing held back.
Every part is in the repository, under Apache-2.0. No account, no tracking, no feature gate.
$npm install -g shadowlapcredentialsKeys pass through.
They go to the provider and are never stored. A clean replay needs none.
127.0.0.1Local only.
The proxy and the dashboard listen on this machine, behind a session token.
.shadowlap/Yours to read.
Recordings stay in your project, readable only by you and kept out of git.
redactMasked on the way out.
Your rules run on every pack, export and share link. A push copies runs as recorded, keys masked.
unpackShared runs are checked.
Every file is verified before anything is written, and replaying one asks before running its command.
telemetryOff by default.
Nothing is sent unless you point it at a collector you run.
A run leaves your machine only when you export, share or push it, or ask a model about it. Report a vulnerability →
How is this different from LangSmith, Langfuse or Braintrust?
They record what happened so you can read and score it. shadowlap re-runs your real code against a recording, finds the call that lost the run and proves the fix. A Langfuse or LangSmith trace imports as a recording, and runs export to Langfuse and Braintrust.
And from replay-based evals like Kitaru?
They ask the model again on every replay, so every replay spends tokens and starts from the top. shadowlap replays the model too: free until the call that changed, which forks with a diff.
And from agent debuggers like Laminar or OrcaReplay?
They replay the model's recorded answers too, up to a call you choose, but run the tools again. shadowlap replays the tools as well, forks at the first request your change alters, fences writes, and adds blame, verify and offline CI.
The model isn't deterministic. How can a replay be exact?
The model isn't asked again: its answers come from the recording, byte for byte. After a fork, verify replays each run until a gate can tell a real fix from a lucky run.
How sure is a verified fix?
Verify replays failing and unseen runs until it tells a 90% pass rate from 70%, and good runs may not drop 10 points. Each verdict is wrong at most 5% of the time, and every figure is yours to set.
What does finding and proving a fix cost?
Only the calls after a fork run live: everything before it replays free, and a run your change doesn't touch costs nothing. --budget limits what blame, repair and verify spend, and each report says what it spent.
My agent sends emails and issues refunds. Is replaying it safe?
Yes, for the tools you wrap: after a fork, write tools are blocked, or stubbed with a value you choose. A replay won't start if the run called an unwrapped tool that could write.
How do you test prompt injection?
shadowlap plants a canary in an untrusted read, in a plain instruction or inside your red-teaming tool's payloads, and replays with writes fenced. A tool call that carries the canary proves the planted text caused it. It complements red teaming; it doesn't replace it.
We have to switch models by a retirement date. What changes?
Run shadowlap migrate on your goldens before the date. It shows each decision the new model makes differently, changed writes first, and checks live only the runs that write, stop or are judged differently. The report ends with what still passes, what it costs and what to set.
Does it work with chat agents that wait for a user?
Yes. What the user types is recorded like a tool's result, so a conversation replays, tests and blames without anyone typing.
Do I have to change my agent's code?
Not for model calls: shadowlap run points your SDK at its proxy. Each tool is wrapped once, in a line or by your framework's adapter, and shadowlap init finds them. MCP servers need none.
Do you train on our runs?
No. Nothing is collected by default and runs are never pooled; they stay in your project. Whether you train on your recordings, or on pairs you export, is between you and your model provider.
Will everything stay free?
Yes. Everything in shadowlap, the team store in your own bucket included, is free under Apache-2.0, and what is free stays free.
Every bad run becomes the lap your fix has to beat.
Reproduce it exactly. Prove the fix. Keep it fixed.
$npm install -g shadowlap