Skip to main content

Documentation Index

Fetch the complete documentation index at: https://handbook.fiddler.ai/llms.txt

Use this file to discover all available pages before exploring further.

Monitor custom AI agents and multi-framework agentic applications with Fiddler using OpenTelemetry’s native instrumentation.

​
What You’ll Learn

In this guide, you’ll learn how to:
  • Set up OpenTelemetry tracing for custom agent frameworks
  • Configure Fiddler as your OTLP endpoint with proper authentication
  • Map agent attributes to Fiddler’s semantic conventions
  • Create instrumented LLM and tool spans with required attributes
  • Verify traces in the Fiddler dashboard
Time to complete: ~10-15 minutes
When to Use Raw OpenTelemetryThis guide is for advanced scenarios requiring full manual OTLP control:
  • Multi-framework environments requiring unified observability across different agent frameworks
  • Existing OpenTelemetry infrastructure where you want to route Fiddler traces through your own OTel pipeline
  • Advanced control over trace sampling, batch processing, and attribute mapping
When to Use Fiddler SDKs Instead (recommended for most users):SDKs provide automatic instrumentation and require significantly less code. Use raw OpenTelemetry only when SDKs don’t fit your use case.

​
Prerequisites

Before you begin, ensure you have:
  • Fiddler Account: An active account with a GenAI application created
  • Python 3.10+
  • OpenTelemetry Packages:
    • pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
  • LLM Provider (for examples): OpenAI API key or similar
  • Fiddler Access Token: Get your token from Settings > Credentials
For a complete working example with advanced patterns, download the Advanced OpenTelemetry Notebook from GitHub or open it in Google Colab.
1

Create Fiddler Application

  1. Log in to your Fiddler instance and navigate to GenAI Applications
  2. Select “Add Application” to create a new application
  3. Copy your Application ID - This must be a valid UUID4 format (e.g., 550e8400-e29b-41d4-a716-446655440000)
  4. Get your Access Token from Settings > Credentials
Important: Keep your Application ID and Access Token secure. You’ll need both for the next steps.
2

Configure Environment Variables

Set up your environment to connect to Fiddler’s OTLP endpoint:
export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-instance.fiddler.ai"
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer <YOUR_ACCESS_TOKEN>,fiddler-application-id=<YOUR_APPLICATION_UUID>"
export OTEL_RESOURCE_ATTRIBUTES="application.id=<YOUR_APPLICATION_UUID>"
Environment Variable Breakdown:
VariableDescriptionExample
OTEL_EXPORTER_OTLP_ENDPOINTYour Fiddler instance URLhttps://org.fiddler.ai
OTEL_EXPORTER_OTLP_HEADERSAuthentication and app ID headersauthorization=Bearer sk-...,fiddler-application-id=550e8400...
OTEL_RESOURCE_ATTRIBUTESResource-level application identifierapplication.id=550e8400-e29b-41d4-a716-446655440000
Python Configuration (alternative to environment variables):
import os

os.environ['OTEL_EXPORTER_OTLP_ENDPOINT'] = 'https://your-instance.fiddler.ai'
os.environ['OTEL_EXPORTER_OTLP_HEADERS'] = 'authorization=Bearer <TOKEN>,fiddler-application-id=<UUID>'
os.environ['OTEL_RESOURCE_ATTRIBUTES'] = 'application.id=<UUID>'
Tip: Store credentials in a .env file and use python-dotenv for local development:
from dotenv import load_dotenv
load_dotenv()  # Loads variables from .env file
3

Initialize OpenTelemetry

Set up OpenTelemetry with Fiddler’s OTLP exporter:
import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

# Initialize tracer provider
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)

# Configure OTLP exporter for Fiddler
otlp_endpoint = os.getenv('OTEL_EXPORTER_OTLP_ENDPOINT') + '/v1/traces'
otlp_exporter = OTLPSpanExporter(endpoint=otlp_endpoint)

# Add batch span processor
otlp_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(otlp_processor)

print(f"✅ OpenTelemetry configured with endpoint: {otlp_endpoint}")
What This Does:
  • TracerProvider: Manages trace generation
  • OTLPSpanExporter: Exports spans to Fiddler via OTLP protocol
  • BatchSpanProcessor: Batches spans for efficient network transmission
Local Debugging: Add a console exporter to see traces locally while developing:
from opentelemetry.sdk.trace.export import ConsoleSpanExporter

console_exporter = ConsoleSpanExporter()
console_processor = BatchSpanProcessor(console_exporter)
trace.get_tracer_provider().add_span_processor(console_processor)
4

Instrument Your Agent

Create instrumented spans for your agent’s operations. Fiddler requires specific attributes to properly categorize and visualize your agent traces.Required Fiddler AttributesResource Level (set via environment variable):
  • application.id - UUID4 of your Fiddler application
Span Level (required for each span):
  • fiddler.span.type - Type of operation: "chain", "tool", "llm", or "agent"
Example: Simplified Travel Agent
import json
from openai import OpenAI

client = OpenAI()

AGENT_NAME = "travel_agent"
AGENT_ID = "travel_agent_v1"

# Define tools
def book_hotel_tool(city: str, date: str):
    """Book a hotel in the specified city."""
    with tracer.start_as_current_span("book_hotel") as span:
        span.set_attribute("fiddler.span.type", "tool")
        span.set_attribute("gen_ai.agent.name", AGENT_NAME)  # optional
        span.set_attribute("gen_ai.agent.id", AGENT_ID)      # optional

        # Tool-specific attributes
        span.set_attribute("gen_ai.tool.name", "book_hotel")
        tool_input = {"city": city, "date": date}
        span.set_attribute("gen_ai.tool.input", json.dumps(tool_input))

        # Execute tool
        result = {"status": "confirmed", "hotel": f"Grand Hotel {city}", "confirmation": "HTL123"}
        span.set_attribute("gen_ai.tool.output", json.dumps(result))

        return result

def book_flight_tool(source: str, destination: str, date: str):
    """Book a flight between two cities."""
    with tracer.start_as_current_span("book_flight") as span:
        span.set_attribute("fiddler.span.type", "tool")
        span.set_attribute("gen_ai.agent.name", AGENT_NAME)  # optional
        span.set_attribute("gen_ai.agent.id", AGENT_ID)      # optional

        # Tool-specific attributes
        span.set_attribute("gen_ai.tool.name", "book_flight")
        tool_input = {"source": source, "destination": destination, "date": date}
        span.set_attribute("gen_ai.tool.input", json.dumps(tool_input))

        # Execute tool
        result = {"status": "confirmed", "flight": "FL456", "departure": "10:00 AM"}
        span.set_attribute("gen_ai.tool.output", json.dumps(result))

        return result

# Agent implementation
def travel_agent(user_request: str):
    """Main travel agent function."""
    with tracer.start_as_current_span("travel_agent_chain") as root_span:
        root_span.set_attribute("fiddler.span.type", "chain")
        root_span.set_attribute("gen_ai.agent.name", AGENT_NAME)  # optional
        root_span.set_attribute("gen_ai.agent.id", AGENT_ID)      # optional

        # Call LLM to understand request
        with tracer.start_as_current_span("llm_call") as llm_span:
            llm_span.set_attribute("fiddler.span.type", "llm")
            llm_span.set_attribute("gen_ai.agent.name", AGENT_NAME)  # optional
            llm_span.set_attribute("gen_ai.agent.id", AGENT_ID)      # optional

            # LLM-specific attributes
            llm_span.set_attribute("gen_ai.request.model", "gpt-4o-mini")
            llm_span.set_attribute("gen_ai.system", "openai")
            llm_span.set_attribute("gen_ai.llm.input.user", user_request)
            llm_span.set_attribute(
                "gen_ai.llm.input.system",
                "You are a travel agent. Parse user requests and call appropriate tools."
            )

            # Call OpenAI
            response = client.chat.completions.create(
                model="gpt-4o-mini",
                messages=[
                    {"role": "system", "content": "You are a travel agent. Parse user requests and call appropriate tools."},
                    {"role": "user", "content": user_request}
                ],
                tools=[
                    {
                        "type": "function",
                        "function": {
                            "name": "book_hotel",
                            "description": "Book a hotel in a city for a specific date",
                            "parameters": {
                                "type": "object",
                                "properties": {
                                    "city": {"type": "string"},
                                    "date": {"type": "string"}
                                },
                                "required": ["city", "date"]
                            }
                        }
                    },
                    {
                        "type": "function",
                        "function": {
                            "name": "book_flight",
                            "description": "Book a flight between two cities",
                            "parameters": {
                                "type": "object",
                                "properties": {
                                    "source": {"type": "string"},
                                    "destination": {"type": "string"},
                                    "date": {"type": "string"}
                                },
                                "required": ["source", "destination", "date"]
                            }
                        }
                    }
                ]
            )

            # Set token usage
            llm_span.set_attribute("gen_ai.usage.input_tokens", response.usage.prompt_tokens)
            llm_span.set_attribute("gen_ai.usage.output_tokens", response.usage.completion_tokens)
            llm_span.set_attribute("gen_ai.usage.total_tokens", response.usage.total_tokens)

            # Process tool calls
            tool_results = []
            if response.choices[0].message.tool_calls:
                for tool_call in response.choices[0].message.tool_calls:
                    tool_name = tool_call.function.name
                    tool_args = json.loads(tool_call.function.arguments)

                    if tool_name == "book_hotel":
                        result = book_hotel_tool(**tool_args)
                        tool_results.append(result)
                    elif tool_name == "book_flight":
                        result = book_flight_tool(**tool_args)
                        tool_results.append(result)

            llm_span.set_attribute("gen_ai.llm.output",
                                 f"Called tools and received: {tool_results}")

        return {"status": "success", "bookings": tool_results}

# Run the agent
result = travel_agent("Book a hotel in Paris for tomorrow and a flight from London to Paris")
print(f"Agent result: {result}")
Key Implementation Details:
  • Chain Spans: Use fiddler.span.type = "chain" for high-level workflows
  • LLM Spans: Include model, system prompt, user input, output, and token usage
  • Tool Spans: Include tool name, input JSON, and output JSON
  • Nested Spans: Create parent-child relationships to show execution flow
Simpler alternative: Install fiddler-otel to skip all the manual OTLP configuration and attribute boilerplate. The SDK’s start_as_current_span() method handles span type enforcement and attribute propagation automatically — no need to set fiddler.span.type, gen_ai.agent.name, or gen_ai.agent.id on every span manually. Typed span wrappers like FiddlerGeneration and FiddlerTool provide helper methods such as set_model() and set_tool_name() instead of raw set_attribute() calls. See Manual Instrumentation in the integration guide.
5

Verify Monitoring

  1. Run your instrumented code using the example above
  2. Wait 1-2 minutes for traces to appear in Fiddler
  3. Navigate to GenAI Applications in your Fiddler instance
  4. Verify application status changes to Active
  5. View traces to see your agent spans, hierarchy, and attributes
Success Criteria:✅ Application shows as Active in GenAI Applications ✅ Traces appear in the trace explorer ✅ Span hierarchy shows chain → LLM → tools relationship ✅ fiddler.span.type is set on every span ✅ LLM token usage is tracked ✅ Tool inputs and outputs are captured
Verification Tip: Check the trace timeline view to see the execution flow of your agent, including which tools were called and how long each operation took.

​
Attribute Reference

​
Required Attributes

Resource Level:
AttributeTypeDescriptionExample
application.idstringUUID4 of your Fiddler application"550e8400-e29b-41d4-a716-446655440000"
Span Level:
AttributeTypeDescriptionValid Values
fiddler.span.typestringType of operation"chain", "tool", "llm", "agent"

​
Optional Attributes

Agent Identification:
AttributeTypeDescriptionExample
gen_ai.agent.namestringName of the AI agent"travel_agent"
gen_ai.agent.idstringUnique identifier for the agent"travel_agent_v1"
Set agent attributes on every span. gen_ai.agent.name and gen_ai.agent.id are optional, but if you include them, set both on every span within the trace. Fiddler uses these attributes to attribute spans to the correct agent — spans missing these fields will be unattributed even if other spans in the same trace carry them.
Conversation Tracking:
AttributeTypeDescriptionExample
gen_ai.conversation.idstringSession/conversation identifier"conv_123"
LLM Span Attributes:
AttributeTypeDescriptionExample
gen_ai.request.modelstringModel name"gpt-4o-mini", "claude-3-opus"
gen_ai.systemstringLLM provider"openai", "anthropic"
gen_ai.llm.input.systemstringSystem prompt"You are a helpful assistant"
gen_ai.llm.input.userstringUser input"What's the weather?"
gen_ai.llm.outputstringLLM response"The weather is sunny"
gen_ai.usage.input_tokensintInput tokens used42
gen_ai.usage.output_tokensintOutput tokens used28
gen_ai.usage.total_tokensintTotal tokens used70
gen_ai.input.messagesstring (JSON array)Chat history provided to model[{"role": "user", "content": "Hello"}]
gen_ai.output.messagesstring (JSON array)Messages returned by model[{"role": "assistant", "content": "Hi"}]
Tool Span Attributes:
AttributeTypeDescriptionExample
gen_ai.tool.namestringTool/function name"search_database"
gen_ai.tool.inputstringTool input (JSON)"{"query": "hotels"}"
gen_ai.tool.outputstringTool output (JSON)"{"results": [...]}"
Custom User-Defined Attributes:
PatternLevelExample
fiddler.session.user.{key}Trace (all spans)fiddler.session.user.user_id = "usr_123"
fiddler.span.user.{key}Span (individual)fiddler.span.user.department = "sales"

​
Troubleshooting

​
Common Issues

Problem: Application not showing as “Active” Solutions:
  1. Verify environment variables are set correctly
  2. Check that OTEL_EXPORTER_OTLP_ENDPOINT includes your Fiddler instance URL
  3. Ensure OTEL_EXPORTER_OTLP_HEADERS contains valid authorization token and application ID
  4. Add console exporter to verify spans are being generated locally
  5. Check network connectivity: curl -I https://your-instance.fiddler.ai
Problem: ModuleNotFoundError for OpenTelemetry packages Solutions: 
# Install all required packages
pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

# Verify installation
pip list | grep opentelemetry
Problem: Spans not appearing in Fiddler Solutions:
  1. Verify required attributes are set: 
    # Every span MUST have this
span.set_attribute("fiddler.span.type", "llm")  # or "tool", "chain", "agent"
# Optional but recommended
span.set_attribute("gen_ai.agent.name", "your_agent")
span.set_attribute("gen_ai.agent.id", "agent_id")
  2. Check resource attributes: 
    # Verify application.id is set
print(os.getenv('OTEL_RESOURCE_ATTRIBUTES'))
  3. Enable console exporter for debugging: 
    from opentelemetry.sdk.trace.export import ConsoleSpanExporter
console_exporter = ConsoleSpanExporter()
console_processor = BatchSpanProcessor(console_exporter)
trace.get_tracer_provider().add_span_processor(console_processor)
Problem: Authentication errors (401 Unauthorized) Solutions:
  1. Regenerate your access token from Fiddler Settings > Credentials
  2. Verify header format: authorization=Bearer &lt;token&gt;,fiddler-application-id=&lt;uuid&gt;
  3. Ensure no extra spaces in header values
  4. Check token hasn’t expired
Problem: Invalid Application ID error Solutions:
  1. Copy Application ID directly from Fiddler UI
  2. Verify UUID4 format: 550e8400-e29b-41d4-a716-446655440000
  3. Ensure no extra quotes or whitespace

​
Configuration Options

​
Basic Configuration

import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

# Set environment variables
os.environ['OTEL_EXPORTER_OTLP_ENDPOINT'] = 'https://your-instance.fiddler.ai'
os.environ['OTEL_EXPORTER_OTLP_HEADERS'] = 'authorization=Bearer <TOKEN>,fiddler-application-id=<UUID>'
os.environ['OTEL_RESOURCE_ATTRIBUTES'] = 'application.id=<UUID>'

# Initialize tracing
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)

# Configure OTLP exporter
otlp_endpoint = os.getenv('OTEL_EXPORTER_OTLP_ENDPOINT') + '/v1/traces'
otlp_exporter = OTLPSpanExporter(endpoint=otlp_endpoint)
otlp_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(otlp_processor)

​
Advanced Configuration

High-Volume Applications (Batch Processing Tuning): 
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Customize batch processor settings
custom_processor = BatchSpanProcessor(
    otlp_exporter,
    max_queue_size=500,           # Default: 2048
    schedule_delay_millis=500,    # Default: 5000
    max_export_batch_size=50,     # Default: 512
    export_timeout_millis=10000   # Default: 30000
)

trace.get_tracer_provider().add_span_processor(custom_processor)
Environment Variable Configuration: 
# Batch processor environment variables
export OTEL_BSP_MAX_QUEUE_SIZE=500
export OTEL_BSP_SCHEDULE_DELAY_MILLIS=500
export OTEL_BSP_MAX_EXPORT_BATCH_SIZE=50
export OTEL_BSP_EXPORT_TIMEOUT=10000
Sampling for Production (Reduce Volume): 
from opentelemetry.sdk.trace import sampling

# Sample 10% of traces
sampler = sampling.TraceIdRatioBased(0.1)

# Create provider with sampler
provider = TracerProvider(sampler=sampler)
trace.set_tracer_provider(provider)
Compression (Reduce Network Usage): 
from opentelemetry.exporter.otlp.proto.http.trace_exporter import Compression

# Enable gzip compression
otlp_exporter = OTLPSpanExporter(
    endpoint=otlp_endpoint,
    compression=Compression.Gzip
)
Using FiddlerClient Alternative (Simplified Setup):
Deprecation notice: Importing FiddlerClient and other core symbols (trace, get_current_span, FiddlerSpan, FiddlerGeneration, FiddlerChain, FiddlerTool, get_client, set_conversation_id) directly from fiddler_langgraph is deprecated and will be removed in a future release. Import from fiddler_otel instead: from fiddler_otel import FiddlerClient.
If you have fiddler-otel installed, you can use FiddlerClient for simplified setup — it handles OTLP configuration automatically. There are two levels of abstraction:Option 1: Get a pre-configured tracer (use raw OTel spans, but skip OTLP configuration):
from fiddler_otel import FiddlerClient

client = FiddlerClient(
    application_id='<FIDDLER_APPLICATION_ID>',
    api_key='<FIDDLER_API_TOKEN>',
    url='<FIDDLER_URL>'
)

tracer = client.get_tracer()

with tracer.start_as_current_span("my_operation") as span:
    span.set_attribute("fiddler.span.type", "chain")
    # ... rest of your code
Option 2: Use SDK span wrappers (typed helper methods, automatic attribute propagation):
from fiddler_otel import FiddlerClient

client = FiddlerClient(
    application_id='<FIDDLER_APPLICATION_ID>',
    api_key='<FIDDLER_API_TOKEN>',
    url='<FIDDLER_URL>'
)

# Span type and agent attributes are set automatically
with client.start_as_current_span("llm_call", as_type="generation") as gen:
    gen.set_model("gpt-4o")
    gen.set_user_prompt(user_input)
    response = call_llm(user_input)
    gen.set_completion(response.content)
    gen.set_usage(response.usage.prompt_tokens, response.usage.completion_tokens)
Both approaches handle OTLP configuration automatically. For graceful exit (e.g. servers or short scripts), call client.shutdown() (or await client.ashutdown() in async) so buffered spans are sent before the process exits. See Manual Instrumentation for complete span wrapper documentation.

​
Next Steps

Now that you have OpenTelemetry integration working: