Sunday, May 24, 2026

Building True Tracing for Multi-Step AI Pipelines

Debugging traditional software applications relies on call stacks, HTTP status codes, and line by line log traces. Debugging non-deterministic LLM pipelines and multi-agent workflows is more complex. 

When an agent outputs an incorrect response or encounters an execution loop, traditional application logs often fail to explain why. True AI observability requires tracing prompt construction steps, model parameters, retrieved context chunks, tool call inputs/outputs, latency metrics, and exact token costs per step. 

In this post I will detail how to implement OpenTelemetry tracing using the Azure AI Inference SDK and Azure Application Insights within Microsoft Azure AI Foundry.


OpenTelemetry structures tracing by wrapping distinct operations in Spans. Spans form hierarchical trees that track execution flow across distributed microservices and LLM tool executions. 

Python Implementation: End-to-End Tracing 

Below is a complete implementation using OpenTelemetry SDK with Azure Monitor and Azure AI Project clients.

Python
import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from azure.monitor.opentelemetry.exporter import AzureMonitorTraceExporter
from azure.ai.inference import ChatCompletionsClient
from azure.ai.inference.models import SystemMessage, UserMessage
from azure.core.credentials import AzureKeyCredential

# Initialize Telemetry Tracing
connection_string = os.getenv("APPLICATIONINSIGHTS_CONNECTION_STRING")
tracer_provider = TracerProvider()
exporter = AzureMonitorTraceExporter(connection_string=connection_string)
tracer_provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(tracer_provider)

tracer = trace.get_tracer("foundry.ai.pipeline", "1.0.0")

def execute_traced_ai_workflow(user_query: str):
    with tracer.start_as_current_span("ai_workflow_execution") as parent_span:
        parent_span.set_attribute("ai.user_query", user_query)
        
        # Step 1: Trace Context Retrieval
        with tracer.start_as_current_span("rag_context_retrieval") as rag_span:
            retrieved_context = "System metric status: CPU load 88%."
            rag_span.set_attribute("ai.rag.retrieved_chunks_count", 1)
            rag_span.set_attribute("ai.rag.context_length", len(retrieved_context))
        
        # Step 2: Trace LLM Inference Call
        with tracer.start_as_current_span("azure_openai_inference") as llm_span:
            endpoint = os.getenv("AZURE_AI_INFERENCE_ENDPOINT")
            key = os.getenv("AZURE_AI_INFERENCE_CREDENTIAL")
            
            client = ChatCompletionsClient(endpoint=endpoint, credential=AzureKeyCredential(key))
            
            messages = [
                SystemMessage(content=f"Use context: {retrieved_context}"),
                UserMessage(content=user_query)
            ]
            
            llm_span.set_attribute("gen_ai.system", "azure_openai")
            llm_span.set_attribute("gen_ai.request.model", "gpt-4o")
            
            response = client.complete(messages=messages, model="gpt-4o")
            
            # Record Token Usage Attributes
            if hasattr(response, "usage"):
                llm_span.set_attribute("gen_ai.usage.prompt_tokens", response.usage.prompt_tokens)
                llm_span.set_attribute("gen_ai.usage.completion_tokens", response.usage.completion_tokens)
                llm_span.set_attribute("gen_ai.usage.total_tokens", response.usage.total_tokens)
                
            completion_text = response.choices[0].message.content
            llm_span.set_attribute("gen_ai.response.completion", completion_text)
            
            return completion_text

Key OpenTelemetry Semantic Conventions for AI

To ensure traces render cleanly in Azure AI Foundry dashboards, standard OpenTelemetry attribute naming conventions should be maintained:


Attribute Key

Description

Example

gen_ai.system

AI system or provider

azure_openai

gen_ai.request.model

Model deployment name requested

gpt-4o

gen_ai.usage.prompt_tokens

Input tokens consumed

452

gen_ai.usage.completion_tokens

Output tokens generated

128

ai.rag.retrieved_chunks_count

Number of context chunks retrieved

4

 Analyzing Traces for Root Cause Analysis

 With structured spans enabled:

 Identifying Tool Call Bottlenecks: Filter spans by duration to quickly identify API latency bottlenecks versus generation delays.

 Debugging Hallucinations: Inspect rag_context_retrieval spans alongside gen_ai.response.completion to verify whether incorrect outputs stemmed from missing source data or model hallucination.


No comments:

Building True Tracing for Multi-Step AI Pipelines

Debugging traditional software applications relies on call stacks, HTTP status codes, and line by line log traces. Debugging non-determinist...