Attaching feedback to OTel-ingested traces with the LangSmith SDK

Applies to: LangSmith (SaaS and self-hosted), traces ingested over OTLP, Python langsmith SDK (JS notes at the end).
Symptom this solves: POST /feedback returns 202 and the feedback never shows up on the trace in the UI; plus how to make feedback durable across network failures.

TL;DR

1. When you ingest a trace over OTLP, LangSmith does not use the OTel trace_id as the run id. It derives the run id from the 8-byte OTel span_id, right-aligned into a 16-byte UUID (high 8 bytes zeroed).

   • span_id = 9222d40b53eb8eec → run_id = 00000000-0000-0000-9222-d40b53eb8eec

2. Post feedback against that derived run id, not the trace id.

3. Use the SDK's Client.create_feedback(...) and pass trace_id. This routes the feedback through the background batched upload path, which is also the path covered by on-disk failure persistence.

4. For durability across network outages, set LANGSMITH_FAILED_TRACES_DIR (and optionally LANGSMITH_FAILED_TRACES_MAX_MB). Failed uploads are written to disk and can be replayed later.

Context: why feedback orphans on OTel traces

The /feedback endpoint returns two different success codes:

• 200 means the run referenced by run_id exists, so the feedback is attached.

• 202 means the row was stored but it is not attached to any run (an orphan). This is the API telling you the run_id you sent does not match any run.

The usual cause is sending the OTel trace_id (reformatted as a UUID) as the run_id. That value never matches a run, so you get a 202 and the feedback silently floats off on its own.

How LangSmith derives the run id

On OTLP ingest, the server converts the binary OTel span id into a UUID: it zeroes a 16-byte buffer and copies the span bytes into the low bytes. For an 8-byte span id, the span bytes land in the low 8 bytes of the UUID and the high 8 bytes stay zero.

The trace_id of a LangSmith run is the run id of the root span of that trace (also derived from the root span's span_id, not from the OTel trace id). For a root span, run_id == trace_id.

How-to (Python SDK)

Step 1: convert the span id to a run id

import uuid

def otel_span_id_to_run_id(span_hex16: str) -> str:
    """16-hex-char (8-byte) OTel span_id -> LangSmith run_id.

    Span bytes occupy the LOW 8 bytes of the UUID; the high 8 bytes are zero,
    mirroring the conversion on the LangSmith ingest side.
    """
    span_hex16 = span_hex16.lower().removeprefix("0x")
    if len(span_hex16) != 16:
        raise ValueError(f"span_id must be 16 hex chars (8 bytes), got {len(span_hex16)}")
    return str(uuid.UUID(int=int(span_hex16, 16)))

uuid.UUID(int=...) does the zero-padding and the 8-4-4-4-12 formatting in one step.

Step 2: create feedback through the SDK

from langsmith import Client

client = Client()  # reads LANGSMITH_API_KEY / LANGSMITH_ENDPOINT from env

run_id   = otel_span_id_to_run_id("9222d40b53eb8eec")    # span you are rating
trace_id = otel_span_id_to_run_id("<root_span_id_hex>")  # root span of the trace

client.create_feedback(
    run_id,
    key="user-feedback",
    score=1.0,
    trace_id=trace_id,   # routes through the background batched + disk-persisted path
)

If the span you are rating is the root span, pass the same value for both run_id and trace_id.

How the SDK handles failures (and why trace_id matters)

There are two layers of resilience, and which one you get depends on whether you pass trace_id.

1. In-memory background queue (default, when trace_id is set)

• Non-blocking, buffered, batched submission. create_feedback enqueues onto an in-memory background queue and returns immediately. A background thread batches items, compresses them, and ships them over the multipart ingest endpoint.

• Automatic retries with backoff. Transient failures (5xx, 429, connection errors, timeouts) are retried with exponential backoff, respecting Retry-After.

• Graceful flush on exit. The client registers an atexit handler and exposes client.flush(), which drains the queue before the process exits.

• Without trace_id: create_feedback falls back to a synchronous POST /feedback (still retried, but blocking, unbatched, and not eligible for the on-disk persistence below).

2. On-disk persistence of failed uploads (opt-in)

Set these environment variables to make the client persist payloads that fail to upload, so they survive a network outage and can be replayed later:

LANGSMITH_FAILED_TRACES_DIR=/path/to/dir     # enables the feature
LANGSMITH_FAILED_TRACES_MAX_MB=500           # optional cap, default 100 (MB)

Behavior, confirmed in the SDK source:

• When it fires: after the background upload's in-process retries are exhausted (or immediately on a non-retryable error), the client serializes the multipart payload to disk. A 409 Conflict is treated as already-ingested and is not persisted.

• What it covers: the background runs/multipart ingest path. Trace/run data always uses this path. Feedback uses it only when you pass trace_id — another reason to always set it. Synchronous feedback (no trace_id) is a direct POST and is not persisted.

• File format: one self-contained JSON file per failed batch, named trace_<timestamp>_<rand>.json, written owner-only (0600):

  {
    "version": 1,
    "endpoint": "runs/multipart",
    "headers": { "Content-Type": "multipart/form-data; boundary=...",
                 "Content-Encoding": "zstd" },   // present when compressed
    "body_base64": "<base64 of the raw request body>"
  }

• Cap behavior (important): the size limit is a stop-writing budget, not a ring buffer. When the directory is already at or over the cap, new payloads are dropped (with a warning); old files are not evicted. So set a generous MAX_MB with headroom for your worst-case outage, and drain the directory (replay + delete) to free space.

• Replay: Replay is a plain re-POST of each saved envelope (decode body_base64, POST it to <api_url>/<endpoint> with the saved headers plus your API key). A minimal replay script is below.

Replay helper

import base64, glob, json, os, requests

API_URL = os.environ.get("LANGSMITH_ENDPOINT", "https://api.smith.langchain.com").rstrip("/")
API_KEY = os.environ["LANGSMITH_API_KEY"]
DIR     = os.environ["LANGSMITH_FAILED_TRACES_DIR"]

for path in sorted(glob.glob(os.path.join(DIR, "trace_*.json"))):
    with open(path) as f:
        env = json.load(f)
    headers = {**env["headers"], "x-api-key": API_KEY}
    body = base64.b64decode(env["body_base64"])
    resp = requests.post(f"{API_URL}/{env['endpoint']}", data=body, headers=headers, timeout=30)
    if resp.status_code < 300:
        os.remove(path)  # uploaded successfully; free the disk budget
        print(f"replayed + removed {os.path.basename(path)} -> {resp.status_code}")
    else:
        print(f"kept {os.path.basename(path)} -> {resp.status_code}: {resp.text[:200]}")

Pass the saved headers through unchanged. If Content-Encoding: zstd is present, the body is already compressed; do not re-encode or recompress it.

What this does and does not guarantee

• Covers: network outages and server errors where an upload was attempted and failed. Those payloads land on disk and replay cleanly once connectivity returns.

• Does not cover: items still sitting in the in-memory queue that were never attempted when a hard crash (kill -9, OOM, power loss) occurs. Disk persistence triggers on a failed upload, not for queued-but-unsent items. For short-lived processes, call client.flush() before exit.

Verifying it worked

Because create_feedback returns a Feedback object even when the run does not exist (it does not surface the 200-vs-202 distinction), confirm attachment out of band:

from langsmith import Client
from langsmith.utils import LangSmithNotFoundError

client = Client()

# 1. The run must exist (the span actually ingested under this id).
run = client.read_run(run_id)        # raises LangSmithNotFoundError if it does not
print("run name:", run.name)

# 2. The feedback should come back when listing by run id.
fb = list(client.list_feedback(run_ids=[run_id], feedback_key=["user-feedback"]))
print("attached feedback:", [str(f.id) for f in fb])

You can also click into the trace in the UI: the run id in the URL is exactly the value you should be posting feedback against.

Verified behavior

A controlled OTel span exported to LangSmith, then feedback created via the SDK:

otel span_id          = b0931630ed32cd62
derived run_id        = 00000000-0000-0000-b093-1630ed32cd62
read_run(derived)     -> found (name=otel-feedback-verify)
create_feedback(correct run_id)    -> feedback attached (list_feedback returns it)
create_feedback(trace_id-as-uuid)  -> succeeds silently, but read_run 404s -> orphan

Common gotchas

• Posting the trace id instead of the derived run id → 202 + orphan (covered above)

• Left-padding vs right-padding. The span bytes go in the low bytes (right side) of the UUID. 00000000-0000-0000-9222-d40b53eb8eec is correct; 9222d40b-53eb-8eec-0000-000000000000 is wrong.

• Forgetting trace_id → feedback still lands, but synchronously, unbatched, and not covered by on-disk persistence.

• Letting the fallback dir hit its cap → new failed payloads are dropped, not the old ones. Replay and delete regularly; size MAX_MB with headroom.

• Killing a short-lived process before flush → queued-but-unsent feedback is lost (disk persistence only catches failed upload attempts). Call client.flush().

JavaScript SDK

Full parity. The same run-id derivation applies; use client.createFeedback(runId, key, { score, traceId }) and await client.flush() before a short-lived process exits. The same env vars (LANGSMITH_FAILED_TRACES_DIR, LANGSMITH_FAILED_TRACES_MAX_MB, default 100 MB) enable on-disk persistence, and the saved file format (version, endpoint, headers, body_base64) is identical, so the same replay approach works.

References

• Attach user feedback (how-to and create_feedback): https://docs.langchain.com/langsmith/attach-user-feedback

• Feedback data format: https://docs.langchain.com/langsmith/feedback-data-format

• OTel / OTLP tracing into LangSmith: https://docs.langchain.com/langsmith/trace-with-opentelemetry

• Client-side feedback without exposing your API key (presigned tokens): https://docs.langchain.com/langsmith/presigned-feedback-tokens