claude-code-observability-harness

Claude Code Observability Harness

Install a self-contained, container-free telemetry stack into a project's .claude/harness/observability/ directory, then instrument the application code so it emits OpenTelemetry traces, metrics, and logs to that stack. The whole thing runs from downloaded binaries, auto-starts on session start, and is meant to be queried by the agent during development — telemetry as a feedback loop, not just a dashboard.

This skill is an extraction of two production harnesses (an event-bus library and an agent web app), so it separates what's universal (the transport plumbing) from what's project-specific (the domain instruments). The instrumentation step is a scan-and-propose, never a blind copy — different apps need different spans and metrics.

Architecture

app (OTel SDK) ──OTLP/HTTP:4318──▶ Vector ──▶ JSONL files          (always)
                                         ├──▶ VictoriaLogs :9428    (full mode)
                                         └──▶ VictoriaMetrics :8428 (full mode)

Two modes — pick one before scaffolding

Default to lite unless the user wants queryable time-series/log search or names VictoriaLogs/VictoriaMetrics. Lite can be upgraded to full later by re-running install in full mode and swapping vector.toml — say so rather than implying lite is permanent.

Workflow

Do these in order. Steps 1–2 and 4–7 are mechanical; step 3 is the judgment call and the reason this skill exists.

1. Confirm mode and detect the project shape

• Decide lite vs full (above). Interactive sessions: if unsure, ask the user in one line. Unattended sessions (compost:implement, or any run with no one to answer): never ask; default to lite and record the choice where the run records its decisions: a comment on the issue when there is one, otherwise the PR description or the final report.
• Find the project root (git rev-parse --show-toplevel), the package manager (prefer uv if pyproject.toml exists), and the language. These templates target Python; for other languages, adapt the OTel SDK calls but keep the Vector/script layer verbatim (it is language-agnostic).
• Pick a SERVICE_NAME (default: the repo/package name). This becomes the OTel service.name resource attribute and the log/metric stream label.
• Detection contract: bash .claude/harness/observability/status.sh --json is how any tool (including this skill on re-runs) checks for an existing harness — it emits {"installed", "running", "mode", "service", "services"}. File absent → no harness.

2. Scaffold the harness files

Copy the bundled scripts/ and rendered assets/ into .claude/harness/observability/:

• install.sh, start.sh, stop.sh, status.sh, verify.sh — copy verbatim (mode-aware via harness.env).
• harness.env — render from assets/harness.env.template with SERVICE_NAME and OBS_MODE.
• vector.toml — for lite, copy scripts/vector.lite.toml verbatim. For full, render scripts/vector.full.toml (replace {{SERVICE_NAME}} — it labels the VictoriaLogs streams). Vector does no placeholder substitution itself, so render before copying.
• Install the OTel SDK (Python). Without it, the rendered otel.py stays in no-op mode and emits nothing — silently. Add the packages (an otel dependency group keeps them out of the production install path): bash uv add --group otel opentelemetry-sdk opentelemetry-exporter-otlp-proto-http Then the app must install that group (uv sync --group otel) wherever telemetry is wanted.
• For the Python app: render assets/otel.py.template to a real module (replace {{SERVICE_NAME}} and {{MODULE_IMPORT}}, e.g. app.otel). See step 3 for where it lives and what instruments to add.
• Render assets/probe.py.template to .claude/harness/observability/probe.py (same two placeholders) — verify.sh runs it in step 6.
• Optionally render assets/observability.rules.md.template to .claude/rules/observability.md so future sessions get query recipes path-scoped to the harness. Substitute all four placeholders or raw {{...}} will leak into the doc: {{SERVICE_NAME}}, {{OBS_MODE}}, {{OTEL_MODULE_PATH}} (e.g. app/otel.py), and {{SINK_SUMMARY}} (e.g. "dated JSONL files" for lite, or "VictoriaLogs + VictoriaMetrics + JSONL" for full).

Then wire the lifecycle (step 4) and gitignore (step 5).

3. Instrumentation scan — the core of this skill

Do not blind-copy instruments. The two source harnesses instrument completely different things (an event bus counts produced/consumed events; an agent app times tool calls and counts tokens). Read references/instrumentation-scanning.md and follow it: scan the codebase, classify candidate sites into the three point-types (tool/boundary call sites, dispatch/router sites, and LLM/agent result sites), then present a structured proposal and wait for confirmation before editing. Propose service.name, 3–6 domain instruments, and the exact files/lines you'd touch. Apply only what the user confirms — patterns in this project may differ enough that a silent write would be wrong.

Unattended sessions (compost:implement, or any run with no one to answer): there is no one to confirm with. Apply the conservative core of the proposal — the 3–6 instruments with the clearest detection signals, never speculative ones — and post the full proposal (applied and skipped, with one-line rationale each) in the same place, so the human reviews it before merging.

The rendered otel.py ships with the no-op-when-absent pattern (zero cost when the OTel SDK isn't installed) and a stdlib-logging→OTLP bridge. Add the confirmed domain instruments to it, and call configure() once at app startup (FastAPI lifespan, CLI entrypoint, or __main__).

4. Wire the SessionStart hook

Add to .claude/settings.json so the stack auto-starts each session (idempotent):

{
  "hooks": {
    "SessionStart": [
      { "matcher": "startup",
        "hooks": [{ "type": "command",
          "command": "bash .claude/harness/observability/start.sh" }] } ]
  }
}

Merge into any existing hooks.SessionStart array rather than overwriting it.

5. Gitignore the runtime artifacts

The harness generates binaries, PIDs, logs, and telemetry data — none belong in git. Add (or confirm) these entries, using directory-level globs so nested files (e.g. data/jsonl/**/*.jsonl) are covered — a known trap is data/*.json silently missing nested paths:

.claude/harness/observability/bin/
.claude/harness/observability/pids/
.claude/harness/observability/logs/
.claude/harness/observability/data/

6. Verify end-to-end (do not skip — empty data dirs are the #1 false "done")

Running the stack proves nothing until telemetry actually lands. Verify in two stages:

1. Transport (scripted): bash bash .claude/harness/observability/verify.sh --py "uv run python" This runs the rendered probe.py (asserts configure()==True — a False means the OTel SDK isn't installed and everything is silently no-op, the real cause behind most "empty dirs"), force-flushes, then polls the JSONL sinks until the probe's span, metric, and log all appear. Non-zero exit names the broken stage.
2. Real call sites: exercise the app once (one request / one CLI run) and confirm the domain instruments fired with real labels (e.g. tool.latency{tool=...}), not just the probe's synthetic ones — use the recipes in references/querying.md.

Keep probe.py — it lives in the harness dir (committed, unlike the gitignored runtime artifacts) and verify.sh is re-runnable whenever the pipeline is in doubt. In full mode, also confirm VictoriaLogs/VictoriaMetrics ingested via references/querying.md.

7. Hand off to the workflow

A harness nobody queries returns nothing. When the repo uses compost (docs/agents/ exists), run compost:setup's observability step so docs/agents/observability.md records this harness; compost:diagnose, compost:verify, and compost:build read that file and query or extend the telemetry at fixed points. Without compost, render the rules file from step 2.

Gotchas (learned the hard way)

• Logging bridge has two gates. A LoggingHandler set to INFO still captures nothing if the root logger is left at its WARNING default — records are filtered before reaching the handler. configure() must lower the root level (the template does). Symptom: traces and metrics flow but the logs sink stays empty.
• CWD matters for relative sink paths. vector.toml writes to relative data/jsonl/... paths; start.sh cds to the harness dir so they resolve. Don't run Vector from elsewhere.
• from otel import tracer captures a stale no-op. Import the module and access otel.tracer / otel.meter as attributes, so callers pick up the real providers after configure() reassigns them. The template documents this.
• Empty data dirs ≠ broken pipeline. Two common benign causes, check in this order: (1) the OTel SDK isn't installed, so configure() returned False and everything no-ops silently — assert configure()==True; (2) the app simply hasn't been exercised since instrumentation. Verify (step 6) before concluding the transport is broken.
• Don't overwrite an existing harness. If the target already has .claude/harness/observability/, inspect it first — it may be more advanced than this template. Surface the diff; never clobber.

Bundled resources

• scripts/ — the stack: install.sh, start.sh, stop.sh, status.sh (with the --json integration contract), verify.sh, and both vector.*.toml variants. Language-agnostic; copy verbatim.
• assets/otel.py.template — Python OTel module: no-op-when-absent, log bridge, {{SERVICE_NAME}} and a marked block for domain instruments.
• assets/probe.py.template — one-shot pipeline probe run by verify.sh.
• assets/harness.env.template — per-project SERVICE_NAME / OBS_MODE.
• assets/observability.rules.md.template — path-scoped query recipes for future sessions.
• references/instrumentation-scanning.md — read before step 3. Detection signals for the three instrumentation point-types and the propose-then-apply workflow.
• references/querying.md — LogsQL/PromQL recipes (full mode) and jq-over-JSONL recipes (lite mode), plus real-time Monitor patterns.

Provenance

Extracted from two production harnesses — a full-stack event-bus library (VictoriaLogs + VictoriaMetrics + transforms + buffers) and a lightweight agent web app (JSONL-only with a stdlib-logging→OTLP bridge). The lite and full modes are those two real variants, so both are battle-tested rather than speculative; the gotchas above are bugs that actually bit.