Agent Observability and Evals Basics: What to Trace, What to Test and What Never to Log

Agent observability and evals come down to two habits. First, record one trace per request, with a span for every model call, every tool call and every handoff. Second, score those traces two ways. Run a small fixed set of cases offline before each release, and sample live traces online to catch what the set missed.

The OpenTelemetry GenAI semantic conventions give you standard names for those spans. As read on November 12, 2026, their status line still says Development, so pin versions and expect renames. The same spec says instrumentations should not capture prompts and outputs by default. Anthropic's January 2026 guide says 20 to 50 cases drawn from real failures is a good first eval set.

The short version

Disclosure. Oakgen is our product, an AI creative studio. Observability is infrastructure, not an Oakgen feature, and nothing here claims Oakgen implements a particular convention.

Who this is for

You have built or are about to ship an agent, and "it feels worse since the last change" is the only signal you have. You might be a developer wiring up a tool-calling agent, or a creative operations lead who wants to know what an agent did with a brief and a budget.

You do not need a platform to start. You need trace IDs, a short list of things never to log, and 20 test cases.

How we checked this

On November 12, 2026 we opened the OpenTelemetry GenAI semantic conventions repository, the LangSmith documentation, the Microsoft Agent Framework observability page, the Google ADK observability pages, Anthropic's evals article, OpenAI's trace grading guide and the LangChain Workday case study.

We did not instrument an agent for this article, and we did not run the eval table below against a real system. The thresholds in it are our starting values, not standards. Everything about a vendor's product comes from that vendor's own pages, and we say so where it matters.

What a trace of an agent looks like

A trace is the whole job, a span is one step inside it. LangSmith's concepts page defines a trace as a collection of runs for a single operation, and says that if you know OpenTelemetry you can think of a run as a span.

An agent trace has a root and children. The OpenTelemetry blog of May 14, 2026 (written by a Microsoft engineer) describes the typical tree. A top-level invoke_agent span, child chat spans for each model call, and execute_tool spans for each tool invocation.

The three vendors we read describe nearly the same tree.

• OpenTelemetry. The spec defines gen_ai.operation.name values including chat, execute_tool, invoke_agent, invoke_workflow, plan and retrieval. It says to use invoke_agent when a model drives control flow through tool calls, and invoke_workflow when the application defines the control flow, such as a graph.
• Microsoft Agent Framework. Its page lists invoke_agent <agent_name> as the top-level span, chat <model_name> for model calls and execute_tool <function_name> for tools. It also lists a token-usage metric and an operation-duration metric.
• Google ADK. Its traces page lists invoke_agent, invoke_workflow, execute_tool and generate_content spans, and says ADK follows the OpenTelemetry GenAI conventions.

LangSmith takes a different route. It has its own run model and also ingests OpenTelemetry spans through an OTLP endpoint, per its Trace with OpenTelemetry page.

How to record a handoff

Handoffs are the step most teams forget to trace. The OpenTelemetry non-normative examples model a handoff as an execute_tool span for the transfer tool, such as transfer_to_agent_b. It carries gen_ai.transfer.target.name and gen_ai.transfer.mode. Two mode values are listed. pass_control, where the target takes over, and return_to_caller, where the source resumes.

The target agent then gets its own invoke_agent span in the same trace. That gives you one trace ID across every agent, which is the property you cannot rebuild afterward. If you want patterns for when to hand off at all, see our post on agent orchestration patterns.

Where the OpenTelemetry conventions stand

The status is Development. We opened the agent spans page, the spans page and the execute-tool page in the semantic-conventions-genai repository on November 12, 2026. Each carries a Development status line. The GenAI attributes we read on them are marked Development too, including gen_ai.operation.name, gen_ai.tool.name and the transfer attributes. Only shared attributes borrowed from elsewhere, such as error.type, are marked Stable. The repository README says it covers GenAI clients, MCP and provider-specific conventions.

The old home has moved. The page at opentelemetry.io/docs/specs/semconv/gen-ai/ now says the conventions moved to that repository and is no longer maintained in the main one.

Third-party sources disagree about stability. Some articles in our search results describe the GenAI conventions as stabilizing or stable. We found no stable marking on the spec pages themselves, so we treat them as unstable.

The consequence is visible in Microsoft's documentation. Agent Framework uses the latest experimental conventions when OTEL_SEMCONV_STABILITY_OPT_IN is unset. If you set that variable without the gen_ai_latest_experimental token, you get the older v1.36 conventions, which emit gen_ai.system where the latest mode emits gen_ai.provider.name. Its default source name begins with Experimental. Same framework, two attribute names for the same idea.

What to log without leaking data

Telemetry is a second copy of your data, with different access rules. Treat content as sensitive by default.

The OpenTelemetry text says model instructions, user messages and outputs are sensitive and often large. It says instrumentations SHOULD NOT capture them by default and SHOULD offer an opt-in. It describes three patterns.

1. Do not record instructions, inputs or outputs. This is the default.
2. Record them on the spans, which suits pre-production or storage that complies with your privacy rules.
3. Store content externally and record references on the span, which the spec recommends for production because external storage allows separate access controls.

The spec's content configuration, when supplied through an environment variable, must be named OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT. Its values are NO_CONTENT (the default), SPAN_ONLY, EVENT_ONLY and SPAN_AND_EVENT. The list of governed attributes includes tool call arguments and results, retrieval query text and memory records, not just chat messages. That matters, because tool arguments are where an email address or an API key tends to hide.

The vendors line up with this.

• Microsoft. ENABLE_SENSITIVE_DATA defaults to false. Its page says to enable it only in development or testing, because prompts, responses, function arguments and results may expose user information in production traces.
• Google ADK. Its page shows a Kotlin setting, TelemetryConfig.captureMessageContent, with the comment "use with caution in production". We did not find an equivalent switch documented on the pages we read for the other languages, so check yours.
• LangSmith. Its sensitive-data page names LANGSMITH_HIDE_INPUTS and LANGSMITH_HIDE_OUTPUTS, plus anonymizer hooks, for stripping content before it leaves your process.

A practical allow list

For a creative team the lesson is concrete. Log the ID of the reference image, not the image. Log the model name and the credit cost, not the unreleased campaign copy. Keep a short, written retention period for the opt-in store.

Offline evals and online monitoring

They answer different questions, and you need both. LangSmith's documentation draws the line cleanly. Offline evaluation runs on a dataset of examples, which have reference outputs, before you ship. Online evaluation runs on real runs and threads from production, which have no reference outputs.

That difference decides what each can check.

Anthropic's article makes the same point from the other side. It lists automated evals, production monitoring, user feedback, manual transcript review and human studies as methods that catch different failures, and compares them to the Swiss cheese model. No layer catches everything.

The loop that makes this worth doing is short. A failing production trace becomes a dataset case. A new evaluator targets that failure. You fix it, run offline to confirm, and redeploy. LangSmith's documentation describes this feedback loop, and OpenAI's trace grading guide describes grading a trace, the end-to-end log of decisions, tool calls and reasoning steps, to find where an agent went right or wrong.

Score the outcome, not the path

Anthropic distinguishes the transcript (the full record of a trial) from the outcome (the final state in the environment). Its example is a flight-booking agent that says the flight is booked. What counts is whether a reservation exists in the database.

It also warns against checking an exact order of tool calls. Its words are that this is "too rigid" and brittle, because agents find valid approaches the designer did not anticipate. Check the end state, check the things that must never happen, and read transcripts to confirm your graders are fair.

One run proves little

Agents vary between runs, so run each case more than once. Anthropic defines pass@k as at least one success in k attempts, and pass^k as all k attempts succeeding. Its example is a 75% per-trial success rate across three trials, which gives about 42% for passing all three.

Do the arithmetic for your own agent. A case that passes 90% of the time passes three runs in a row 73% of the time (0.9 x 0.9 x 0.9 = 0.729). A 95% case passes ten in a row about 60% of the time. Use pass^k for anything customer-facing, where users expect it to work every time.

Asset 1. A minimum viable observability checklist

This is the smallest set we would ship with. Each row has a test you can run in an afternoon.

Cost caps pair with agent orchestration cost control, and the extra spans multi-agent systems add are covered in single agent versus multi-agent.

Asset 2. A starter eval-case table

Start here. Each row is a type of case, with how to score it and a pass rule. Write three to five concrete cases per row from your own failures, and you have your first 20 to 50.

The pass rules are our starting values. Tighten them as the agent matures, and lower them only with a written reason. Where a row says "code," a plain function can grade it.

Three rules for using the table.

• Balance the set. Anthropic's article warns that one-sided evals produce one-sided optimization. Row 4 exists so you do not breed an agent that calls a tool for everything.
• Prove each case is solvable. Write a reference solution that passes every grader. A 0% pass rate on a case across many trials, Anthropic says, most often signals a broken task, not a weak agent.
• Split capability from regression. Capability cases should start at a low pass rate and give you something to climb. Regression cases should sit near 100% and block releases. Anthropic describes graduating capability cases into the regression set once they pass reliably.

A vendor-reported case, labeled as one

LangChain's case study of Workday, dated November 10, 2026, is a vendor-published customer story, so read it as such. It says Workday's four-person Agent Platform team used LangGraph for orchestration and LangSmith for tracing, observability and evals. It reports prototype time falling from four to five months to under a day, and the team growing from four people to about 30.

What it does not say matters more. It gives no eval pass rates, no trace volumes and no breakdown of how much of the speedup came from tracing or evals versus the agent server and SDK. The speedup is also credited to the platform as a whole. We would not cite it as evidence that observability makes teams faster. It is evidence that a large enterprise team treats tracing and evals as part of its platform from the start, per its vendor.

How a creative team would apply this

This is the one place we will tie it to our own product, and the point is the practice, not a feature claim. An agent that plans a campaign, calls image and video tools and spends credits needs the same few records. Which brief started the trace, which tools ran, what each step cost, and where a human approved. Rows 5, 7 and 12 of the eval table matter most for a team spending money.

We cover where a person should approve in what a creative AI agent should automate, a way to score an agent on a brief in the 100-point creative agent scorecard, and how a pipeline chains steps in agentic creative pipelines. For how model checks work behind a generation product, see how Oakgen picks and checks models. If your tool calls go through a gateway, that is one more place to log, which our gateway explainer covers.

Common mistakes

1. Logging full prompts everywhere. It is the fastest way to turn telemetry into a privacy incident. Default to no content.
2. Forgetting tool arguments. The spec treats arguments and results as content, and so should you.
3. Tracing model calls but not handoffs. You then cannot say which agent owned a bad answer.
4. Grading the path. Exact tool-call sequences break whenever the agent finds a valid route you did not predict.
5. Running each case once. One pass tells you little. Use at least three trials for anything users depend on.
6. Building the dataset from imagination. Start from your bug list and support queue, as Anthropic advises.
7. Trusting an LLM judge without checking it. Calibrate it against human labels, and give it an "Unknown" option.
8. Hard-wiring attribute names. The conventions are in Development. Keep your own fields.
9. Setting alerts on everything. Three alerts with owners beat thirty that nobody reads.
10. Never reading transcripts. Anthropic says it does not take an eval score at face value until someone reads some transcripts.

Decision framework

Match the setup to your stage.

• Prototype, one agent, one developer. Console traces from your framework, a spreadsheet of 20 cases, and a script that runs them three times. Content capture on, locally only.
• Pre-launch. An OpenTelemetry tracer and a backend, the checklist above, 30 to 50 cases in CI, and content capture off in production.
• In production. Add online evals on a sample, an external store for opt-in content with access control, a failing-trace-to-dataset habit and the three alerts.
• Multiple agents or many teams. Add shared span naming and a mapping layer so a convention change is a one-file edit.

On tools, a framework that already emits OpenTelemetry spans (Microsoft Agent Framework and ADK both document this) lets you choose the backend later. LangSmith is a reasonable pick if you want tracing, datasets and online evaluators in one product and accept a vendor dependency. We did not benchmark the backends against each other. Our comparison of frameworks is in Microsoft Agent Framework, Google ADK and LangGraph.

What I would do

I would do four things in this order, in the first week.

1. Add a trace ID that survives every handoff, plus agent and prompt version on every span.
2. Turn content capture off in production and write down the retention period.
3. Write 25 cases from real failures, using the table above, and run each three times before every release.
4. Block a release on any approval-gate or injection violation, and on a regression case that drops.

I would not buy a platform before step three. I would not wait for the conventions to go stable either. Use the names now, keep a mapping layer, and move on.

To see which image, video and audio models an agent can call in Oakgen, browse the model library.

Frequently asked questions

What is agent observability?

It is the ability to reconstruct what an agent did from the telemetry it left behind. In practice that means one trace per user request, with a span for each model call, each tool call and each handoff, plus token counts, latency, errors and the identity of the agent and model. Google's ADK documentation says basic input and output monitoring is typically insufficient for agents with any significant complexity, because the failure is usually in a middle step. Observability is what lets you debug that step. Evals are the separate practice of scoring behavior against defined criteria.

Are the OpenTelemetry GenAI semantic conventions stable?

No, not as of November 12, 2026. The spans, execute-tool and agent-span pages in the OpenTelemetry GenAI semantic conventions repository carry a status of Development, and the GenAI attributes we opened on those pages are marked Development as well. The old opentelemetry.io page now says the conventions moved to the semantic-conventions-genai repository. Some third-party posts describe them as stable or nearly so, but the spec pages themselves do not. Pin your instrumentation version and expect attribute names to change.

Should I log full prompts and model outputs?

Not by default. The OpenTelemetry text says instrumentations SHOULD NOT capture instructions, inputs and outputs by default and SHOULD offer an opt-in. It lists three patterns. Record nothing (the default), record content on spans, or store content externally and record references on the span, which it recommends for production. Microsoft Agent Framework defaults ENABLE_SENSITIVE_DATA to false and warns to enable it only in development or testing. Log identifiers, sizes, token counts and tool names instead, and keep content in a store with its own access rules and retention.

What is the difference between offline and online evals?

Offline evals run your agent against a curated dataset before release, and the cases have reference outputs, so you can check correctness and compare versions. Online evals score real production traces after the fact. LangSmith's documentation says online evaluators have no reference outputs, so they check quality patterns, safety and format, and you can apply filters and sampling rates to control cost. Use offline evals to block a bad release and online evals to catch what your dataset missed, then feed the failing traces back into the dataset.

How many eval cases do I need to start?

Fewer than most teams think. Anthropic's January 2026 article on agent evals says 20 to 50 simple tasks drawn from real failures is a great start, because early changes have large effects that small samples can detect. It adds that mature agents need larger and harder sets. Start from what you test by hand before each release, add every user-reported failure, and include cases where the agent should not act, so you do not optimize toward an agent that does everything.

Do I need LangSmith to do this?

No. The pieces are an OpenTelemetry-compatible tracer, a place to store traces, a dataset of cases and a runner that grades them. LangSmith documents OpenTelemetry ingestion and offers tracing and evals as a product, and Microsoft Agent Framework and Google ADK emit OpenTelemetry spans that any compatible backend can take. A script that runs 30 cases and checks results with plain code is a valid first eval harness. Buy a platform when reading traces and managing datasets by hand becomes the bottleneck.

Sources

1. OpenTelemetry GenAI semantic conventions, agent spans page (Status Development), read 2026-11-12
2. OpenTelemetry GenAI semantic conventions, spans and content capture page, read 2026-11-12
3. OpenTelemetry GenAI semantic conventions, execute tool page, read 2026-11-12
4. OpenTelemetry GenAI semantic conventions, agent call examples (non-normative), read 2026-11-12
5. OpenTelemetry, Moved: Generative AI semantic conventions (opentelemetry.io notice), read 2026-11-12
6. OpenTelemetry blog, Inside the LLM Call: GenAI Observability with OpenTelemetry, May 14 2026 (read 2026-11-12)
7. LangSmith docs, Observability concepts (read 2026-11-12)
8. LangSmith docs, Evaluation concepts, offline and online evaluation (read 2026-11-12)
9. LangSmith docs, Trace with OpenTelemetry (read 2026-11-12)
10. LangSmith docs, Prevent logging of sensitive data in traces (read 2026-11-12)
11. Microsoft Learn, Agent Framework observability, updated Nov 10 2026 (read 2026-11-12)
12. Google ADK docs, Observability for agents (read 2026-11-12)
13. Google ADK docs, Traces (read 2026-11-12)
14. Anthropic, Demystifying evals for AI agents, Jan 9 2026 (read 2026-11-12)
15. OpenAI, Trace grading guide (read 2026-11-12)
16. LangChain, How Workday Built an Enterprise-Scale AI Agent Platform with LangGraph and LangSmith, Nov 10 2026 (vendor case study, read 2026-11-12)