Token burn is the new CPU/memory
In traditional application infrastructure, the primary production health signals are CPU, memory, latency, and error rate. For AI workloads, there’s a fifth signal that most teams don’t track until costs become a problem: token consumption. Input tokens, output tokens, cost per request, cost per workflow, cache hit rate. These signals are the economic heartbeat of any AI system, and they behave very differently from compute costs — they scale with content complexity, not just with traffic volume.
A document-processing workflow that runs 50,000 pages through an LLM for summarization can cost $40,000 in a single month, or $8,000 with the right model selection and prompt optimization. Both produce output. The business doesn’t know which it’s running until the invoice arrives, if it doesn’t have token-level telemetry on every workflow.
An audit log is not observability. Observability is: can you see which workload is running what model, at what cost, with what quality signal, against what baseline? If those four questions don’t have answers you can pull up in two minutes, you don’t have governance — you have paperwork.
The standardization layer that actually exists
OpenTelemetry’s GenAI semantic conventions have matured into a usable standard. Spans now have defined fields for prompts and completions, tool call chains, model version and provider, input and output token counts, and agent reasoning steps. This means there’s a common instrumentation language that works across Anthropic, OpenAI, Google, and open-weight models running on local infrastructure.
The practical implication: a team that instruments their AI workloads with OpenTelemetry gets a unified trace that shows the entire request journey — user request to inference to tool call to response — in the same observability backend they already use for application monitoring.
What useful AI observability actually looks like
The maturity progression is relatively clear. At the basic level, you have token counts and latency per request — a proxy-level intercept that takes minutes to add and immediately surfaces cost anomalies. At the working level, you have full distributed traces, cost attribution per workload, and basic quality evaluation. At the advanced level, you have hierarchical agent tracing, automated quality evaluation running continuously on production traffic, and drift detection that catches when a model’s output distribution shifts over time.
Where this connects to governance
Governance without telemetry is a statement of intent, not a control. The organizations that are ahead on this have figured out that the policy layer (what may we do?) and the technical layer (what are we actually doing?) have to be connected. A vendor-level audit log from each AI provider is not enough — it doesn’t span providers, doesn’t attribute cost to business decisions, and doesn’t surface quality signals. A unified telemetry layer does all three, and it’s the foundation any meaningful AI governance program has to build on before the policy discussion becomes real.