Agent O11y

observability + deterministic re-execution

Re-execute the failure that shipped.

Agent O11y captures every real execution of your agent — spans, dependency calls, model responses — as OpenTelemetry. Then it re-executes the original artifact against recorded dependency responses, so a fix can be proven against the exact conditions that failed.

OTel-native ingest · replay-grade SDKs on top

See the loop See the debugger 
$ pip install agent-o11y
$ export AGENT_O11Y_ENDPOINT=https://ingest.example
$ export AGENT_O11Y_ENVIRONMENT=prod

Capture is a side channel — it never blocks, changes, or breaks the request. What each variable does is documented in the SDK READMEs.

a run’s verdict is one of PASS FAIL INCONCLUSIVE NOT_EVALUATED with a reason code — FIXTURE_MISSING, never an impression a policy miss = INCONCLUSIVE — never a live write

A 200 response is not a correct agent.

An agent can return a successful HTTP response while choosing the wrong tool, using stale context, misreading a value, or falsely claiming that an action completed. Engineers investigate these failures after prompts, data, configuration, and deployments have changed — and reconstruct the environment manually, test the fix informally, and fail to preserve the investigation as a regression.

The product captures the evidence and the executable inputs required to turn that investigation into a controlled experiment — then into a durable test.

What did it actually do?

Every span, dependency call, and model response, recorded as OpenTelemetry — not inferred from logs afterward.

Can it happen again?

The original artifact re-executes under the recorded fixtures — the same conditions, with production side effects blocked.

Will the fix survive the next release?

Approved behavior is compiled into immutable regression revisions that run in CI.

The loop.

production execution → case → reproduction → candidate experiment → approved contract → regression revision → ci

  1. capture

    What did the agent actually do?

    Instrument the entry point and its dependencies with the Python, Go, or TypeScript SDK. Every model-visible input, tool call, and response becomes an OpenTelemetry span and a versioned execution event. Capture is fail-open — it never blocks, changes, or breaks the request.

  2. replay

    Can it happen again?

    The original artifact — an OCI image by digest — re-executes under a run-scoped dependency gateway. Recorded fixtures answer the dependency calls; production side effects are blocked; budgets and policies are enforced at the gateway.

  3. diff

    Where did it diverge?

    Live and replayed runs are compared event by event. Matching is strict, on canonical request digests: a miss creates a divergence record and stops the run as INCONCLUSIVE — never a silent live call.

  4. review

    What behavior is correct?

    Contract review and assertions pin the intended behavior. The verdict is PASS, FAIL, INCONCLUSIVE, or NOT_EVALUATED — with a reason code such as FIXTURE_MISSING.

  5. ship

    Did the fix survive the release?

    Approved behavior is compiled into an immutable revision — offline, deterministic regression gating: no live calls, no flaky, INCONCLUSIVE on divergence. Experiments pin baseline and candidate to one fixture-set revision, so every changed factor is named. The suite runs from CI.

The debugger, running.

The product on two seeded tenants — an analytics agent and a sales agent, switchable from the org picker in the top bar. The same views you get when you self-host the stack.

Overview of the sales tenant: open items include the case where a stale 2026-Q2 price tier was quoted for a 2026-Q3 contract, findings from the order sweep, failure signals, and per-producer health
overview — the open items name the failure: a stale 2026-Q2 price tier quoted for a 2026-Q3 contract
The sales agent's execution corpus: fourteen captured runs from Meridian Industrial with origin, environment, duration, and error count; sidebar lists Overview, Cases, Executions, Experiments, Regressions, Releases, Review, Learning
the sales agent's execution corpus — one row per captured run; exec_s941 is the 296-span order sweep
Execution detail of the order sweep: PASS verdict, replay mode PLAYBACK on a fixture set, flame strip, 296-span waterfall tree, and per-component breakdown
execution detail — flame strip, waterfall, per-component breakdown; mode PLAYBACK, 296 spans, 144 of 200 budgeted calls

The platform, end to end.

One Go control plane, one web app, one lifecycle — and the twelve surfaces in the app’s sidebar are the entire product:

  • overview
  • cases
  • executions
  • experiments
  • regressions
  • releases
  • review
  • learning
  • integrations
  • organization
  • usage & billing
  • settings
ingest
Three SDKs — Python, Go, TypeScript — share one wire format. Or speak plain OTLP/HTTP if you already emit OpenTelemetry.
executions
One row per production request: filter by environment and origin, search, sort — then open the detail view with its flame strip, waterfall, per-event payloads, and per-component breakdown.
replay
Original-artifact re-execution under an immutable replay contract. Per-call dependency policies — recorded, scenario, sandbox, blocked — with live modes disabled in ordinary replay and CI.
divergence
Strict matching on canonical request digests. A mismatch is a divergence record; the run ends INCONCLUSIVE. Missing evidence never becomes an implicit passing assertion.
experiments
Baseline and candidate runs pinned to one fixture-set revision, so every changed factor is named in the comparison.
regressions
Frozen, runnable case revisions and the suites that gate releases.
ci
Approved behavior runs from CI without a human repairing it by hand — the saved investigation is the test.

Three SDKs, one protocol.

All three SDKs record dependency calls at replay grade — not just spans. If you already emit OpenTelemetry, plain OTLP/HTTP is the door in.

python pip install agent-o11y stdlib-only core · python 3.11+
go go get github.com/agent-o11y/sdk-go stdlib only · go 1.24+
typescript npm install @agent-o11y/sdk zero runtime deps · node 20+ · esm
open telemetry OTLP/HTTP any exporter · traces, metrics, logs · json or protobuf

Why replay, not a re-run?

Re-running the agent hits live tools, live data, and live models. It reproduces the shape of a failure, not the failure itself — and it may do things you cannot undo.

The original artifact re-executes under a run-scoped dependency gateway. The gateway answers each dependency call with its recorded response, so the run compares behavior, not infrastructure. A fixture miss stops the run as INCONCLUSIVE; the platform would rather tell you it doesn’t know than answer from a live endpoint mid-run.

The invariants.

  • Production side effects are blocked in ordinary replay and CI.
  • A fixture is never matched by tool name alone.
  • Strict fixture matching; no silent fallback.
  • Missing evidence never becomes an implicit passing assertion.
  • A run can finish successfully while its quality verdict is inconclusive or failing.
  • Deterministic exactness is stated only relative to a declared scope and comparison projection.

Your agents already leave evidence.

Make it replayable. Instrument one entry point, capture a real failure, and replay it before the context is gone.

Walk the loop See the SDKs