/docs

Send your first trace.

TwoTail ingests standard OpenTelemetry. There is no SDK to install: create an API key, send OTLP/JSON to the ingest endpoint, and your traces are live. About 10 minutes. Looking for what connects? See Integrations.

last updated 2026-07-10 view as markdown
1

Get an API key

Create a key in the TwoTail app under API Keys. Keys start with tt_live_ and the full key is shown once, at creation, so copy it somewhere safe. Creating and revoking keys requires an owner, admin, or developer role on your account.

No account yet? Book a demo and we'll set you up.

2

Send traces

Two paths. Most teams use the first.

Path A · Your coding agent does it

Copy the prompt below into Claude Code, Cursor, or any coding agent working on your codebase. It instruments your agent end to end: trace structure, sessions, LLM and tool attributes, evals. Replace <TWOTAIL_API_KEY> with your key (or tell your agent where to find it), then review the diff.

raw markdown →
View the full prompt
Add TwoTail tracing to this agent. TwoTail accepts standard OpenTelemetry spans - no custom conventions required.

## Configuration
- Endpoint: https://www.twotail.ai/api/v1/traces
- API Key: <TWOTAIL_API_KEY> (created in the TwoTail app under API Keys)
- Auth Header: X-API-Key

## Trace Structure (important)
Generate ONE trace_id per agent run and use the SAME trace_id for ALL spans in that run. Each span gets its own unique span_id. Link child spans to parents via parentSpanId. This groups everything into a single trace so you can see the full execution flow.
- trace_id: 32 lowercase hex chars (16 bytes) - e.g., "4bf92f3577b34da6a3ce929d0e0e4736"
- span_id: 16 lowercase hex chars (8 bytes) - e.g., "00f067aa0ba902b7"
- parentSpanId: same format as span_id, or null/omit for root spans

## Sessions (multi-turn conversations)
If your agent runs multi-turn conversations, set gen_ai.conversation.id on the spans of each turn to the conversation/thread id. Use one trace_id per turn (run) and the SAME gen_ai.conversation.id across the whole conversation. TwoTail groups all traces sharing a gen_ai.conversation.id into one session.

## Standard OTel Attributes (recommended - the more you send, the more TwoTail can analyze)

For LLM calls:
- gen_ai.system: "openai" | "anthropic" | etc (auto-detects span as LLM type)
- gen_ai.request.model: model name (e.g., "gpt-4", "claude-3-sonnet")
- gen_ai.prompt: the input prompt/messages (important for debugging and analysis)
- gen_ai.completion: the output completion (important for debugging and analysis)
- gen_ai.usage.input_tokens: input token count
- gen_ai.usage.output_tokens: output token count

For tool calls:
- tool.name: name of the tool being called (auto-detects span as tool type)
- tool.parameters: input parameters (JSON)
- tool.result: output result

For evaluations (as child spans of the span being evaluated):
- Start the span name with "eval." or "eval_" (e.g., "eval.relevance", "eval_toxicity") - auto-detects as evaluation
- Set parentSpanId to the span being evaluated
- If using an LLM for evaluation, include gen_ai.* attributes (will be typed as 'llm')
- Recommended attributes (latest OTel GenAI eval semconv):
  - gen_ai.evaluation.name: name of the evaluation (e.g., "relevance", "toxicity")
  - gen_ai.evaluation.score.value: numeric score (e.g., 0-1 or 1-5)
  - gen_ai.evaluation.score.label: human-readable label (e.g., "pass", "relevant")
  - gen_ai.evaluation.explanation: explanation text from the evaluator
  - (legacy eval.name / eval.score / eval.passed / eval.reason are still accepted)

## Business Attributes
Include custom attributes that identify the business entities your agent works with. Any attributes you add are preserved in metadata and can be queried in TwoTail. Examples:
- user.id, account.id, gen_ai.conversation.id (conversation/session id for chatbots and multi-turn agents)
- document.id, document.url (for RAG agents)
- target.url (for web browsing/fetch agents)
- repo.name, pull_request.id (for coding agents)
- task.type, workflow.name (for productivity and automation agents)

## OTel OTLP/JSON Format
{
  "resourceSpans": [{
    "resource": {
      "attributes": [
        {"key": "service.name", "value": {"stringValue": "my-agent"}}
      ]
    },
    "scopeSpans": [{
      "spans": [{
        "traceId": "abc123...",
        "spanId": "def456...",
        "parentSpanId": null,
        "name": "chat_completion",
        "startTimeUnixNano": 1234567890000000000,
        "endTimeUnixNano": 1234567890100000000,
        "attributes": [
          {"key": "gen_ai.system", "value": {"stringValue": "openai"}},
          {"key": "gen_ai.request.model", "value": {"stringValue": "gpt-4"}},
          {"key": "gen_ai.usage.input_tokens", "value": {"intValue": 150}},
          {"key": "gen_ai.usage.output_tokens", "value": {"intValue": 50}}
        ],
        "status": {"code": 1}
      }]
    }]
  }]
}

## Implementation Notes
- status.code: 0=UNSET, 1=OK, 2=ERROR
- Timestamps are integer Unix time in NANOseconds
- Batch spans and send at most 1,000 per request
- Handle HTTP errors gracefully (don't break the agent if tracing fails)

Prefer redacted payloads? Signed-in users can generate redacted and call-only variants of this prompt in the app, under Data > Integration Guide.

Path B · Test the endpoint with curl

Send one span by hand to confirm your key works. Swap in a fresh timestamp so the trace shows up at the top of your recent traces.

curl -X POST https://www.twotail.ai/api/v1/traces \
  -H "Content-Type: application/json" \
  -H "X-API-Key: <TWOTAIL_API_KEY>" \
  -d '{
    "resourceSpans": [{
      "resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "test-agent"}}]},
      "scopeSpans": [{
        "spans": [{
          "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
          "spanId": "00f067aa0ba902b7",
          "name": "test_llm_call",
          "startTimeUnixNano": 1783977600000000000,
          "endTimeUnixNano": 1783977600100000000,
          "attributes": [
            {"key": "gen_ai.system", "value": {"stringValue": "openai"}},
            {"key": "gen_ai.request.model", "value": {"stringValue": "gpt-4"}}
          ],
          "status": {"code": 1}
        }]
      }]
    }]
  }'
3

Verify

A successful request returns:

{
  "success": true,
  "message": "Successfully ingested 1 spans from OTel format",
  "spans_inserted": 1
}

Open the app and your trace is on the Data page, under Traces. Spans that fail to parse are skipped rather than rejected: the request still succeeds and the skip count usually shows in message. spans_inserted is the authoritative count, so check it matches what you sent.

Not seeing your trace?

  • 401 Unauthorized. The X-API-Key header is missing, or the key is wrong, revoked, or expired. Keys must start with tt_live_.
  • Response says success but spans are missing. Malformed spans are skipped, not rejected. spans_inserted is the authoritative count: compare it to what you sent, and look for "(N malformed spans skipped)" in message.
  • Trace appears with a 1970 date, or not in recent traces. Timestamps are integer Unix time in nanoseconds. Millisecond or second values are accepted without error but land decades in the past.

Limits

  • Maximum 1,000 spans per request. Batch below that.
  • OTLP/JSON only: no protobuf, no gRPC, no gzip. Protobuf-only exporters can route through an OpenTelemetry Collector with JSON encoding.
  • Accounts pending approval are capped at 10,000 stored spans. Contact support@twotail.ai to lift the limit.

Spot a problem on this page? Email us.