Link metrics to traces (exemplars)¶

An exemplar attaches a trace id to a single metric sample — not a label — so a latency spike on a Prometheus/Grafana dashboard links straight to the trace that produced it. You get the drill-down with zero cardinality cost, because the trace id never becomes a label.

transport-metrics records exemplars on every duration histogram that observes through the shared Metrics.Observe path — http_request_duration_seconds, grpc_server_handling_seconds, operation_duration_seconds, db_query_duration_seconds and queue_processing_seconds — whenever an exemplar source is configured. queue_wait_seconds is the one exception: ObserveWait takes no context, so there is no trace id to read.

Enable it¶

The trace-id source is pluggable so the core needs no tracing dependency. Use the opt-in metrics/otel subpackage to read the current OpenTelemetry span:

import (
    metrics "gitlab.com/phpboyscout/go/transport-metrics"
    metricsotel "gitlab.com/phpboyscout/go/transport-metrics/otel"
)

m, err := metrics.New(
    metrics.WithExemplarSource(metricsotel.ExemplarSource()),
)

Now every duration observation made on a context carrying a span gets a trace_id exemplar. metricsotel.ExemplarSource() reads only the span context — it starts no spans and needs no tracer provider — so it is safe whether or not tracing is configured; it yields nothing when the context carries no span.

It checks for the presence of a trace id, not for the sampling decision, so an unsampled span still produces an exemplar. That exemplar points at a trace your backend most likely never stored, so following it from a dashboard will dead-end. If that matters, wrap the source and drop unsampled spans yourself:

otelSource := metricsotel.ExemplarSource()

metrics.WithExemplarSource(func(ctx context.Context) (string, bool) {
    if !trace.SpanContextFromContext(ctx).IsSampled() {
        return "", false
    }
    return otelSource(ctx)
})

A custom source¶

Not on OpenTelemetry? Supply your own — anything that pulls a trace id from the request context:

metrics.WithExemplarSource(func(ctx context.Context) (string, bool) {
    if id := myTraceID(ctx); id != "" {
        return id, true
    }
    return "", false
})

Scraping exemplars¶

Exemplars ride on the OpenMetrics exposition format. The handler offers it, but only serves it when the client asks — point a scraper that supports exemplars (Prometheus with --enable-feature=exemplar-storage, or an OTLP-scraping agent) at /metrics, and wire Grafana's exemplar-to-trace link on the trace_id label.

Why curl shows no exemplars¶

curl sends no Accept header, so it gets the Prometheus text format back — which has nowhere to put an exemplar. The samples are being recorded; the format is hiding them. Ask for OpenMetrics explicitly:

curl -s -H 'Accept: application/openmetrics-text; version=1.0.0' \
  localhost:8080/metrics | grep trace_id

http_request_duration_seconds_bucket{method="GET",route="GET /x",status="200",le="0.005"} 1 # {trace_id="0102030405060708090a0b0c0d0e0f10"} 2.2e-07 1.7856969705221822e+09

The exemplar is the part after the #: the trace id, the observed value, and the timestamp. If that comes back empty, the source is not yielding a trace id — check there is a span on the context at the point the observation is made, not just somewhere upstream.

The bridge to OTel, not a conflict¶

Exemplars are the deliberate meeting point between this Prometheus (pull) module and the OTel (push) go/observability modules: your metrics stay in Prometheus, your traces stay in your tracing backend, and an exemplar stitches a sample to a trace across the two. It complements the OTel metrics story — it does not compete with it.