Agent Observability & Evaluation

Core Concepts

Foundational principles of telemetry and testing for compound agent loops.

OpenTelemetry Spans

Instrument agent steps and helper tools using OTel tracers to build complete call-chain graphs (inspecting latency, prompt, and response attributes).

OTLP Log Collectors

Configure unified OpenTelemetry protocol (OTLP) exporters to stream build logs, telemetry data, and system traces to remote servers.

LLM-Assisted Metrics

Run evaluations (G-Eval, hallucination scoring, toxic tone assessments) by employing dedicated evaluator models to review agent logs.

Metric Gating Thresholds

Set minimum compliance scores (e.g. hallucination score < 0.3) that code must satisfy to progress past deployment gates.

Cost & Latency Tracking

Log token expenses and API latency attributes at step boundaries, identifying slow tools or expensive prompt architectures.

Guardrails

Intercept execution threads dynamically to block PII output leaks, prompt injection jailbreaks, or unsafe tool behaviors.

Setup & Installation

Configure tracing backends and python testing suites.

Install Packages

                    # Install Arize Phoenix OpenTelemetry tracing client

                    pip install arize-phoenix opentelemetry-sdk opentelemetry-api opentelemetry-exporter-otlp

                    # Install DeepEval library for testing suites

                    pip install deepeval

Instrumentation & Evaluation

Registering OTel trace providers, exporting spans, and setting up multi-metric tests.

                    from deepeval import assert_test

                    from deepeval.test_case import LLMTestCase

                    from deepeval.metrics import HallucinationMetric, AnswerRelevancyMetric

                    from phoenix.otel import register

                    from opentelemetry import trace

                    # 1. Initialize Phoenix Tracing with OTLP Exporter support

                    register(endpoint="http://localhost:6006/v1/traces")

                    tracer = trace.get_tracer(__name__)

                    # 2. Trace step execution block injecting span properties

                    async def run_traced_tool(query: str):

                        with tracer.start_as_current_span("query_database_tool") as span:

                            span.set_attribute("db.query", query)

                            result = "Constraint Duplicate Key failure in database registry."

                            span.set_attribute("db.result", result)

                            return result

                    # 3. Build DeepEval Gated Test Case running multiple metrics

                    def test_agent_compliance():

                        test_case = LLMTestCase(

                            input="Why did deploy task tx_992 fail?",

                            actual_output="Deploy task failed due to duplication primary keys.",

                            retrieval_context=["Deploy logs warning: task tx_992 failed on DB constraint duplicate primary key."]

                        )

                        hallucination = HallucinationMetric(threshold=0.3)

                        relevancy = AnswerRelevancyMetric(threshold=0.7)

                        # Gated execution: fails test if metrics cross defined thresholds

                        assert_test(test_case, [hallucination, relevancy])

Frequently Asked Questions

Is OpenTelemetry required for agent tracing?▶

It is not strictly required, but it is the open-source industry standard. Setting up OTel instrumentation guarantees you can export trace data to any visualization tool (like Arize Phoenix, Langfuse, or Datadog) without changing your core agent code later.

How do we model evaluations (DeepEval metrics) without calling OpenAI?▶

DeepEval allows you to configure local models (e.g. running Llama-3 locally via Ollama or vLLM) as the evaluator agent, letting you run continuous regression tests locally or in sandboxed CI/CD runners without any network dependencies.

What is OpenTelemetry (OTel) and why is it standard for agents?▶

OpenTelemetry is an open-source standard for logs, metrics, and traces. It is essential for AI agents because it wraps non-linear reasoning, tool calls, and LLM completions inside standardized nesting spans, decoupling your codebase from proprietary monitoring providers.

How do I evaluate agent performance over time?▶

By compiling a golden dataset of user queries, expected responses, and contexts. When you release new prompts or model changes, run your evaluation metrics (like Answer Relevancy or Hallucination) over the entire golden set to identify regressions.

What are LLM-assisted metrics?▶

LLM-assisted metrics (like DeepEval's G-Eval or RAGAS Faithfulness) use a powerful evaluator model (like GPT-4) to grade a target agent's response based on scoring instructions and context, rather than relying on exact word-matching calculations.

How do I set up a local Arize Phoenix collector?▶

Simply execute `python -m phoenix.server.main` in your terminal environment. It spins up a local telemetry collector on port 6006, ready to receive incoming OTLP trace payloads from your instrumented python scripts.