Telemetry

The liminis-context-graph service emits structured JSON Lines (JSONL) telemetry events that give operators per-call timing, token usage with estimated cost, and WAL throughput counters.

Transport

Default: Events are written to stderr, one JSON object per line.

{"type":"ipc_call","ts_ms":1716100000000,"method":"knowledge_find_entities","request_id":1,"duration_ms":42,"success":true}

Future: Set LIMINIS_TELEMETRY_SOCKET to a UNIX socket path to stream events there instead of (or in addition to) stderr. This transport is not yet implemented.

To capture events from the default transport:

./liminis-context-graph 2> telemetry.jsonl

Event Types

Telemetry is scoped to the process, not to individual graphs within it: no event type carries a group_id (or similar per-graph) field today, including ipc_call for calls that operate on a specific group — this is a gap for anyone trying to attribute telemetry to one graph in a multi-graph workspace, not a documentation omission.

All events share two common fields:

Field	Type	Description
type	string	Discriminant identifying the event kind (see table below)
ts_ms	u64	Unix epoch timestamp in milliseconds when the event was emitted

ipc_call

Emitted after every IPC request completes, from handlers::dispatch(). This is a hot-path event — it fires on every request.

Field	Type	Description
method	string	JSON-RPC method name (e.g. knowledge_add_episode)
request_id	any	JSON-RPC request id value as-is
duration_ms	u64	Wall-clock time from request receipt to response, in milliseconds
success	bool	true if the handler returned Ok, false for any error

Example:

{"type":"ipc_call","ts_ms":1716100000000,"method":"knowledge_add_episode","request_id":1,"duration_ms":42,"success":true}

token_usage

Emitted after every successful Anthropic API call from extractor.rs. This is a hot-path event for knowledge_add_episode calls.

Field	Type	Description
role	string	Which LLM use-case produced these tokens ("extraction", future: "dedup")
model	string	Anthropic model identifier (e.g. claude-haiku-4-5-20251001)
input_tokens	u64	Input tokens billed by the API
output_tokens	u64	Output tokens billed by the API
cache_read_tokens	u64	Tokens served from the prompt cache (cheaper rate)
cache_creation_tokens	u64	Tokens written into the prompt cache
estimated_cost_usd	f64 or null	Estimated cost in USD, or null if the model is not in the pricing table

Example:

{"type":"token_usage","ts_ms":1716100000001,"role":"extraction","model":"claude-haiku-4-5-20251001","input_tokens":512,"output_tokens":128,"cache_read_tokens":384,"cache_creation_tokens":0,"estimated_cost_usd":0.000512}

extraction_truncated

Emitted when do_extract detects a stop_reason: "max_tokens" response and triggers a budget-doubling retry. Emitted once per chunk, after the retry resolves (either with success or with a second budget overflow). If retry_succeeded is false, the chunk was lost and an error was returned to the caller.

Field	Type	Description
ts_ms	integer	Unix timestamp in milliseconds
model	string	Anthropic model identifier that triggered the overflow
chunk_len_bytes	integer	Length of the episode body chunk in bytes
initial_max_tokens	integer	The max_tokens value used for the first (overflowing) attempt
retry_succeeded	bool	true if the doubled-budget retry produced a valid result; false if the retry also overflowed

Example:

{"type":"extraction_truncated","ts_ms":1716100000050,"model":"claude-sonnet-4-6","chunk_len_bytes":12480,"initial_max_tokens":8192,"retry_succeeded":true}

llm_fallback

Emitted when the primary LLM is unavailable and extraction falls back to the secondary model configured via LCG_EXTRACTION_LLM’s primary:fallback form.

Field	Type	Description
role	string	Which LLM use-case triggered the fallback
primary_model	string	Model that failed
fallback_model	string	Model being used instead
error_reason	string	Reason the primary model was unavailable (e.g. "rate_limit_exceeded")

Example:

{"type":"llm_fallback","ts_ms":1716100000002,"role":"extraction","primary_model":"claude-sonnet-4-6","fallback_model":"claude-haiku-4-5-20251001","error_reason":"rate_limit_exceeded"}

wal_append

Emitted after each WAL entry is written. Not yet emitted — pending issue #3 (WAL implementation).

Field	Type	Description
duration_us	u64	Time to append the WAL entry, in microseconds
bytes	integer	Size of the appended WAL entry in bytes

Example:

{"type":"wal_append","ts_ms":1716100000003,"duration_us":180,"bytes":1024}

service_state

Emitted when the daemon changes operational state: on degraded startup, after successful recovery, and during graceful shutdown.

Field	Type	Description
state	string	One of "degraded", "healthy", "shutting_down", or "stopped"
reason	string or absent	Machine-readable reason code (e.g. "lbug_wal_corrupt"). Present when state = "degraded".
detail	JSON value or absent	Structured detail, typically a string carrying the lbug error. Present when state = "degraded"

Degraded example (emitted at startup when lbug WAL is corrupt):

{"type":"service_state","ts_ms":1716523200000,"state":"degraded","reason":"lbug_wal_corrupt","detail":"database error: Lbug(Runtime exception: Corrupted wal file. Read out invalid WAL record type.)"}

Healthy example (emitted after successful knowledge_recover):

{"type":"service_state","ts_ms":1716523260000,"state":"healthy"}

Shutting-down example (emitted at the start of graceful shutdown, before in-flight tasks are drained):

{"type":"service_state","ts_ms":1716523270000,"state":"shutting_down"}

Stopped example (emitted immediately before exit(0), after initiating the WAL checkpoint; if the inner shutdown timeout was exceeded, in-flight tasks may still be winding down and the checkpoint is best-effort):

{"type":"service_state","ts_ms":1716523271000,"state":"stopped"}

The renderer uses this event to update the recovery UI state without polling knowledge_status. On every clean exit, the telemetry stream ends with "shutting_down" → "stopped".

wal_replay_complete

Emitted once when WAL replay finishes — at startup, after knowledge_rebuild_from_wal, and at the end of autonomous recovery.

Field	Type	Description
mutations_replayed	u64	WAL mutations successfully applied
unrecognised_lines	u64	Lines whose shape matched no known mutation template
failed_lines	u64	Lines that parsed but whose statement failed to execute
unparseable_lines	u64	Lines that were not valid JSON
legacy_skipped_lines	u64	Lines skipped as a superseded legacy format
duration_ms	u64	Total replay wall-clock time in milliseconds

A nonzero failed_lines, unrecognised_lines, or unparseable_lines means the rebuilt graph is not a faithful reconstruction of the WAL — each counts a line whose content did not make it into the graph, whether it failed to execute, matched no known template, or was not valid JSON. Use LCG_REPLAY_FAILURE_SAMPLES to surface examples. legacy_skipped_lines is the one counter that does not indicate loss: those lines are a superseded format that is intentionally not replayed.

Example:

{"type":"wal_replay_complete","ts_ms":1716100000004,"mutations_replayed":1284,"unrecognised_lines":0,"failed_lines":0,"unparseable_lines":0,"legacy_skipped_lines":0,"duration_ms":380}

structured_output_parse

Emitted by OaiExtractor for every entity/edge extraction response on the local / OpenAI-compatible path, recording whether the model’s structured output was usable as-is.

Field	Type	Description
model	string	Model that produced the response
call_type	string	"entities" or "edges"
outcome	string	"clean" (valid JSON as returned), "recovered" (needed fence/prefix stripping), "malformed" (not valid JSON at all), or "schema_invalid" (valid JSON that failed schema/field validation on a genuinely required field, ADR-0314)

A high "recovered" or "malformed" rate is the main signal that a local model is a poor fit for extraction. The Anthropic path uses tool-use and does not emit this event.

Example:

{"type":"structured_output_parse","ts_ms":1716100000060,"model":"qwen3.6-27b","call_type":"edges","outcome":"recovered"}

entities_missing_summary

Emitted by OaiExtractor::do_extract_entities on every successful entity-extraction parse that produced at least one entity — OAI-only; the Anthropic path’s tool-use schema doesn’t need this signal and never emits it. Its tool_use schema requires the summary key to be present but not non-empty, so a model can satisfy the schema while still returning entities with no summary text. This event surfaces that missing-summary rate as its own signal, separate from the pass/fail classification in structured_output_parse/extraction_failure: an empty summary is a degraded entity, not a failed extraction. See ADR-0314.

Field	Type	Description
model	string	Model that produced the response
chunk_key	string or null	The episode name (production) or corpus chunk title (lcg-eval), or null
entities_extracted	usize	Total entities parsed from this chunk
missing_summary	usize	Count of those entities whose summary is empty — absent, explicit null, and explicit "" in the source JSON are all indistinguishable after parsing and all count here

Example:

{"type":"entities_missing_summary","ts_ms":1716100000058,"model":"qwen3.6-27b","chunk_key":"notes-0001","entities_extracted":6,"missing_summary":1}

extraction_failure

Emitted at all three extraction-call failure sites (HTTP error, budget exhaustion that persists after one retry, or a malformed/unparseable response) from inside AnthropicExtractor / OaiExtractor. This is the heavier, complete-body sibling of extraction_truncated and structured_output_parse — it carries the full raw response body for forensics, where those two stay lightweight counting-only events. It is consumed only by the sidecar-writing ExtractionFailureSink (see the failure-record sidecar), not by the counting sink ipc_call/token_usage use.

Field	Type	Description
model	string	The model name in force for the failing call
call_type	string	"entities" or "edges"
chunk_key	string or null	The episode name (production) or corpus chunk title (lcg-eval), or null
classification	string	"http_error", "truncation", "malformed" (content that never parsed as JSON at all), or "schema_invalid" (valid JSON that failed schema/field validation, ADR-0314)
raw_body	string	The complete raw response body — never truncated to a prefix. A UTF-8 decoding failure is stored lossily rather than dropping the record. May echo back source-text content from the failing call; apply the same review-before-sharing care as LLM cassettes.
finish_reason	string or null	The provider’s stop/finish reason, or null for an HTTP-level failure
completion_tokens	u64 or null	Output token count, or null if unavailable
max_tokens	u32	The max_tokens value in force for the failing call
entities_extracted	usize or null	Entities already extracted for this chunk before an edge-call failure discarded them from the caller’s return value. Some(count) only at call_type: "edges" failure sites; None at call_type: "entities" sites, where there is nothing to report yet.

Example:

{"type":"extraction_failure","ts_ms":1716100000055,"model":"qwen3.6-27b","call_type":"edges","chunk_key":"notes-0001","classification":"truncation","raw_body":"{\"choices\":[{\"message\":{\"content\":\"[{\\\"predicate\\\"...\"}}]}","finish_reason":"length","completion_tokens":8192,"max_tokens":8192,"entities_extracted":4}

See ADR-0306 for the design rationale, and Testing & Evaluation for the on-disk sidecar file this event’s consumer writes.

wal_rotated

Emitted when the WAL rolls over to a new file.

Field	Type	Description
from_file_seq	u32	Sequence number of the file just closed
to_file_seq	u32	Sequence number of the file now being written
closed_bytes	u64	Size of the closed file in bytes
closed_events	u64	Number of events in the closed file

Example:

{"type":"wal_rotated","ts_ms":1716100000070,"from_file_seq":3,"to_file_seq":4,"closed_bytes":8388608,"closed_events":15220}

workspace_migration

Emitted at each phase of a workspace layout migration.

Field	Type	Description
phase	string	Migration phase reached
detail	JSON value or absent	Structured phase detail

Example:

{"type":"workspace_migration","ts_ms":1716100000080,"phase":"complete"}

wal_auto_recovery

Emitted at each phase of autonomous WAL-corruption self-recovery — the observability for the self-healing path described in Operations. Every field except phase is optional and present only where the phase produces it.

Field	Type	Description
phase	string	One of "corruption_detected", "checkpoint_drop_complete", "cursor_derived", "replay_complete", "index_build_complete", "recovery_complete", "fallback_triggered"
from_seq	u64 or absent	WAL sequence the replay resumed from
cursor_reason	string or absent	How the resume cursor was derived
mutations_replayed	u64 or absent	Mutations applied during recovery replay
elapsed_ms	u64 or absent	Wall-clock time for the phase
fallback_reason	string or absent	Why automatic recovery gave up, on "fallback_triggered"

A stream ending in "recovery_complete" means the service healed itself; one ending in "fallback_triggered" means it needs manual intervention.

Example:

{"type":"wal_auto_recovery","ts_ms":1716100000090,"phase":"replay_complete","from_seq":7,"mutations_replayed":412,"elapsed_ms":1830}

chunk_text_oversized

Emitted by knowledge_process_chunk when chunk_text’s character count exceeds the advisory threshold (default 8000, overridable via LCG_CHUNK_TEXT_ADVISORY_MAX_CHARS — see Configuration). A lightweight, counting-only event mirroring extraction_truncated’s shape, so oversized ingestion is visible in aggregate across a telemetry stream, not just in the single call’s response warning field. Ingestion itself is unaffected — no rejection, truncation, or splitting — this event is purely observational, and fires on every oversized call, including repeated resubmissions of the same chunk_id.

The threshold and this event’s chunk_text_chars field are measured in characters, not bytes.

Field	Type	Description
ts_ms	integer	Unix timestamp in milliseconds
chunk_id	string	The oversized call’s chunk_id
source_file	string	The oversized call’s source_file
chunk_text_chars	integer	Character count of chunk_text (not bytes)
threshold_chars	integer	The advisory threshold in effect for this call

Example:

{"type":"chunk_text_oversized","ts_ms":1716100000095,"chunk_id":"page-042","source_file":"webbrain/page-042.html","chunk_text_chars":21953,"threshold_chars":8000}

---

Sample Output

A complete session ingesting one episode and running one search:

{"type":"ipc_call","ts_ms":1716100000000,"method":"knowledge_build_indices","request_id":1,"duration_ms":12,"success":true}
{"type":"token_usage","ts_ms":1716100000100,"role":"extraction","model":"claude-haiku-4-5-20251001","input_tokens":512,"output_tokens":128,"cache_read_tokens":384,"cache_creation_tokens":0,"estimated_cost_usd":0.000512}
{"type":"ipc_call","ts_ms":1716100000150,"method":"knowledge_add_episode","request_id":2,"duration_ms":320,"success":true}
{"type":"ipc_call","ts_ms":1716100000500,"method":"knowledge_find_entities","request_id":3,"duration_ms":18,"success":true}

---

Pricing Table

Token cost estimates use the compiled-in pricing table at assets/llm_pricing.json. To override at runtime without recompiling:

LIMINIS_LLM_COST_TABLE_PATH=/path/to/my_pricing.json ./liminis-context-graph

The JSON schema matches the built-in table:

{
  "claude-haiku-4-5-20251001": {
    "input_per_mtok": 0.80,
    "output_per_mtok": 4.00,
    "cache_read_per_mtok": 0.08,
    "cache_creation_per_mtok": 1.00
  }
}

Rates are in USD per million tokens. Models not present in the table produce "estimated_cost_usd": null.

---

Environment Variables

Variable	Default	Description
LIMINIS_LLM_COST_TABLE_PATH	(built-in)	Path to a JSON pricing table; overrides the compiled-in defaults
LIMINIS_TELEMETRY_SOCKET	(unset)	UNIX socket path for telemetry output (not yet implemented)
LCG_SHUTDOWN_TIMEOUT_MS	30000	Inner shutdown timeout in milliseconds; process aborts in-flight tasks after this and exits (best-effort WAL checkpoint). Sized to leave headroom under the liminis-app outer budget of 60 s.