Observability

Agentlab integrates with Langfuse for flow execution tracing and observability. This allows flow designers and administrators to monitor how flows execute, identify bottlenecks, and debug issues.

What Gets Traced

When a flow executes, Langfuse captures:

• Flow execution spans — the overall flow run, including timing and status.
• Component-level spans — individual component executions within the flow.
• LLM calls — model invocations including prompts, completions, token usage, and latency.

Prometheus Metrics

Independently of Langfuse, each instance can expose LLM and flow usage as Prometheus metrics. This works with or without Langfuse configured — the two are unrelated.

Enable it with LANGFLOW_PROMETHEUS_ENABLED=true. Metrics are served on LANGFLOW_PROMETHEUS_PORT (default 9090) at /metrics.

What Is Exposed

Metric	Type	Labels
karli_llm_tokens_total	counter	model, kind (input/output), flow_id
karli_llm_calls_total	counter	model, status (ok/error), flow_id
karli_llm_call_duration_seconds	histogram	model, flow_id
karli_flow_runs_total	counter	flow_id, status (ok/error)
karli_flow_duration_seconds	histogram	flow_id
karli_flow_info	gauge	flow_id, flow_name

The _total and _seconds suffixes are appended by the Prometheus exporter, so query the names exactly as listed above. Duration histograms use buckets suited to LLM latency (0.1s to 120s) rather than the OpenTelemetry defaults.

model is the requested model id (for example karli/KARLI-MoE-latest), not the normalized name the gateway returns. When the model cannot be determined it is recorded as unknown rather than dropped.

Token counters are only written when the provider actually reports usage. A call that returns no usage block still increments karli_llm_calls_total, so a gap between call counts and token counts means the provider did not report — not that the call was free. Failed calls are counted with status="error" and contribute no tokens, because a failed call has no authoritative usage.

There are deliberately no session_id, user_id, trace_id or per-component labels, for cardinality reasons. Tokens only — no cost or pricing data is exposed; pricing is applied centrally at read time.

Flow Names

Usage counters are labelled with flow_id only. Flow names arrive through a separate info metric, which always carries the value 1 for a flow's current name:

karli_flow_info{flow_id="…", flow_name="Support Triage"} 1

Join it onto the counters at query time:

sum by (flow_name) (
  increase(karli_llm_tokens_total{kind="output"}[24h])
  * on(flow_id) group_left(flow_name) (karli_flow_info == 1)
)

The == 1 matters. When a flow is renamed the superseded name is set to 0 rather than removed, and group_left fails with "multiple matches for labels" if both are live — so always filter.

Names are kept off the counters on purpose. A label value is part of a time series' identity, so putting the name there would mean a rename starts a brand-new series, strands the old one at its last value, and makes any query spanning the rename wrong. Keying the counters by the immutable flow_id means a rename only moves the info series, and historical graphs immediately re-render under the new name.

A flow appears in karli_flow_info once it has run in the current process.

Instance Identity

Nothing about instance identity is in these metrics. There is no client or instance label. The central Prometheus attaches instance identity through its scrape configuration, sourced from the deployments-API inventory. Do not add such a label in the application.

Limitations

• Single worker. OpenTelemetry metrics are per-process, and only one worker can bind the metrics port. With LANGFLOW_WORKERS greater than 1, only one worker's counters are scraped and the rest are invisible. Keep metrics-enabled instances at one worker until aggregation through a collector is in place.
• Usage metering is intentionally not disabled by LANGFLOW_DEACTIVATE_TRACING. That setting turns off trace export to Langfuse and other providers; metrics keep running.

Accessing Traces

Traces are available through the Langfuse dashboard provided with your Karli Studio deployment.

Tip

When debugging a flow that produces unexpected results, start by reviewing the Langfuse trace for the conversation. The component-level spans show exactly what each component received and produced.