Sunday, May 24, 2026

Building True Tracing for Multi-Step AI Pipelines

Debugging traditional software applications relies on call stacks, HTTP status codes, and line by line log traces. Debugging non-deterministic LLM pipelines and multi-agent workflows is more complex. 

When an agent outputs an incorrect response or encounters an execution loop, traditional application logs often fail to explain why. True AI observability requires tracing prompt construction steps, model parameters, retrieved context chunks, tool call inputs/outputs, latency metrics, and exact token costs per step. 

In this post I will detail how to implement OpenTelemetry tracing using the Azure AI Inference SDK and Azure Application Insights within Microsoft Azure AI Foundry.


OpenTelemetry structures tracing by wrapping distinct operations in Spans. Spans form hierarchical trees that track execution flow across distributed microservices and LLM tool executions. 

Python Implementation: End-to-End Tracing 

Below is a complete implementation using OpenTelemetry SDK with Azure Monitor and Azure AI Project clients.

Python
import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from azure.monitor.opentelemetry.exporter import AzureMonitorTraceExporter
from azure.ai.inference import ChatCompletionsClient
from azure.ai.inference.models import SystemMessage, UserMessage
from azure.core.credentials import AzureKeyCredential

# Initialize Telemetry Tracing
connection_string = os.getenv("APPLICATIONINSIGHTS_CONNECTION_STRING")
tracer_provider = TracerProvider()
exporter = AzureMonitorTraceExporter(connection_string=connection_string)
tracer_provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(tracer_provider)

tracer = trace.get_tracer("foundry.ai.pipeline", "1.0.0")

def execute_traced_ai_workflow(user_query: str):
    with tracer.start_as_current_span("ai_workflow_execution") as parent_span:
        parent_span.set_attribute("ai.user_query", user_query)
        
        # Step 1: Trace Context Retrieval
        with tracer.start_as_current_span("rag_context_retrieval") as rag_span:
            retrieved_context = "System metric status: CPU load 88%."
            rag_span.set_attribute("ai.rag.retrieved_chunks_count", 1)
            rag_span.set_attribute("ai.rag.context_length", len(retrieved_context))
        
        # Step 2: Trace LLM Inference Call
        with tracer.start_as_current_span("azure_openai_inference") as llm_span:
            endpoint = os.getenv("AZURE_AI_INFERENCE_ENDPOINT")
            key = os.getenv("AZURE_AI_INFERENCE_CREDENTIAL")
            
            client = ChatCompletionsClient(endpoint=endpoint, credential=AzureKeyCredential(key))
            
            messages = [
                SystemMessage(content=f"Use context: {retrieved_context}"),
                UserMessage(content=user_query)
            ]
            
            llm_span.set_attribute("gen_ai.system", "azure_openai")
            llm_span.set_attribute("gen_ai.request.model", "gpt-4o")
            
            response = client.complete(messages=messages, model="gpt-4o")
            
            # Record Token Usage Attributes
            if hasattr(response, "usage"):
                llm_span.set_attribute("gen_ai.usage.prompt_tokens", response.usage.prompt_tokens)
                llm_span.set_attribute("gen_ai.usage.completion_tokens", response.usage.completion_tokens)
                llm_span.set_attribute("gen_ai.usage.total_tokens", response.usage.total_tokens)
                
            completion_text = response.choices[0].message.content
            llm_span.set_attribute("gen_ai.response.completion", completion_text)
            
            return completion_text

Key OpenTelemetry Semantic Conventions for AI

To ensure traces render cleanly in Azure AI Foundry dashboards, standard OpenTelemetry attribute naming conventions should be maintained:


Attribute Key

Description

Example

gen_ai.system

AI system or provider

azure_openai

gen_ai.request.model

Model deployment name requested

gpt-4o

gen_ai.usage.prompt_tokens

Input tokens consumed

452

gen_ai.usage.completion_tokens

Output tokens generated

128

ai.rag.retrieved_chunks_count

Number of context chunks retrieved

4

 Analyzing Traces for Root Cause Analysis

 With structured spans enabled:

 Identifying Tool Call Bottlenecks: Filter spans by duration to quickly identify API latency bottlenecks versus generation delays.

 Debugging Hallucinations: Inspect rag_context_retrieval spans alongside gen_ai.response.completion to verify whether incorrect outputs stemmed from missing source data or model hallucination.


Saturday, May 02, 2026

Setting Up Automated Cost Guardrails for LLM Pipelines

Generative AI projects often start with predictable prototyping budgets. However, moving pipelines to production introduces unpredictable user input volumes, unconstrained multi turn conversations, agent loop cascades, and recursive retrieval steps. Without strict runtime cost controls, spending can escalate very quickly.

Managing costs effectively requires proactive, dynamic guardrails embedded into client SDK wrappers and gateway components. This blog post provides technical patterns using Python and TypeScript to implement token caps, cost estimation middleware, request throttling, and model fallback routing in Azure AI Foundry environments.

Anatomy of LLM Cost Drivers

Unexpected billing spikes generally trace back to three root causes:

Prompt Inflation: System instructions and full conversation histories attached to every turn without context truncation.

Agentic Loop Cascades: Autonomous agents hitting edge conditions and looping continuously against model endpoints.

Over-provisioned Model Selection: Routing basic text routing or classification tasks to large models like GPT-4o instead of smaller models like GPT-4o-mini.




Implementing Middleware Token Guardrails
Below is a Python implementation of an Azure AI Inference SDK client wrapped with dynamic token counting, prompt budget enforcement, and fallback model routing.



import sys
import tiktoken
from azure.ai.inference import ChatCompletionsClient
from azure.ai.inference.models import SystemMessage, UserMessage
from azure.core.credentials import AzureKeyCredential

class CostGuardedFoundryClient:
    def __init__(self, primary_endpoint: str, primary_key: str, fallback_endpoint: str, fallback_key: str):
        self.primary_client = ChatCompletionsClient(
            endpoint=primary_endpoint, credential=AzureKeyCredential(primary_key)
        )
        self.fallback_client = ChatCompletionsClient(
            endpoint=fallback_endpoint, credential=AzureKeyCredential(fallback_key)
        )
        self.tokenizer = tiktoken.get_encoding("cl100k_base")
        
        # Pricing constants per 1k tokens (example estimation rates)
        self.PRIMARY_INPUT_COST = 0.0025
        self.FALLBACK_INPUT_COST = 0.00015
        self.MAX_ALLOWED_PROMPT_TOKENS = 4000

    def _count_tokens(self, text: str) -> int:
        return len(self.tokenizer.encode(text))

    def generate_guarded_completion(self, system_prompt: str, user_prompt: str, max_cost_limit_usd: float):
        total_input_tokens = self._count_tokens(system_prompt) + self._count_tokens(user_prompt)
        
        # Guardrail 1: Enforce Token Truncation
        if total_input_tokens > self.MAX_ALLOWED_PROMPT_TOKENS:
            raise ValueError(f"Prompt token count ({total_input_tokens}) exceeds maximum limit ({self.MAX_ALLOWED_PROMPT_TOKENS}).")

        # Guardrail 2: Dynamic Routing Based on Budget & Token Count
        estimated_primary_cost = (total_input_tokens / 1000.0) * self.PRIMARY_INPUT_COST
        
        if estimated_primary_cost > max_cost_limit_usd:
            # Fallback to lower-cost model deployment
            print(f"Primary model estimated cost (${estimated_primary_cost:.4f}) exceeds target limit (${max_cost_limit_usd:.4f}). Routing to fallback model.")
            response = self.fallback_client.complete(
                messages=[
                    SystemMessage(content=system_prompt),
                    UserMessage(content=user_prompt)
                ],
                model="gpt-4o-mini"
            )
        else:
            response = self.primary_client.complete(
                messages=[
                    SystemMessage(content=system_prompt),
                    UserMessage(content=user_prompt)
                ],
                model="gpt-4o"
            )
            
        return response

TypeScript Implementation for Real Time Prompt Truncation

In Node.js applications interacting with Microsoft Azure AI Foundry, sliding-window message truncation prevents conversation histories from growing unbounded.

TypeScript
import { AzureKeyCredential } from "@azure/core-auth";
import ModelClient from "@azure-rest/ai-inference";

interface ChatMessage {
  role: "system" | "user" | "assistant";
  content: string;
}

export class TokenBudgetManager {
  private maxTokens: number;

  constructor(maxTokens: number = 3000) {
    this.maxTokens = maxTokens;
  }

  // Basic token estimation heuristic (approx 4 chars per token)
  private estimateTokens(text: string): int {
    return Math.ceil(text.length / 4);
  }

  public truncateHistory(systemPrompt: string, history: ChatMessage[]): ChatMessage[] {
    let currentTokens = this.estimateTokens(systemPrompt);
    const truncated: ChatMessage[] = [];

    for (let i = history.length - 1; i >= 0; i--) {
      const msgTokens = this.estimateTokens(history[i].content);
      if (currentTokens + msgTokens > this.maxTokens) {
        break;
      }
      currentTokens += msgTokens;
      truncated.unshift(history[i]);
    }

    return [{ role: "system", content: systemPrompt }, ...truncated];
  }
}


Architectural Cost Controls

Circuit Breakers: Implement rate limits using Azure API Management (APIM) placed in front of Azure OpenAI endpoints. Set quota keys by subscription ID or Entra ID user principal.

Semantic Caching: Store prompt-response pairs in Azure Cache for Redis to avoid executing duplicate queries against model endpoints.

Building True Tracing for Multi-Step AI Pipelines

Debugging traditional software applications relies on call stacks, HTTP status codes, and line by line log traces. Debugging non-determinist...