Link metrics to traces (exemplars)¶

An exemplar attaches a trace id to a single metric sample, not to a label, so a latency spike on a Prometheus/Grafana dashboard links straight to the trace that produced it. You get the drill-down with zero cardinality cost, because the trace id never becomes a label.

transport-metrics records exemplars on every duration histogram that observes through the shared Metrics.Observe path (http_request_duration_seconds, grpc_server_handling_seconds, operation_duration_seconds, db_query_duration_seconds and queue_processing_seconds) whenever an exemplar source is configured. queue_wait_seconds is the one exception: ObserveWait takes no context, so there is no trace id to read.

Enable it¶

The trace-id source is pluggable so the core needs no tracing dependency. Use the opt-in metrics/otel subpackage to read the current OpenTelemetry span:

import (
    metrics "gitlab.com/phpboyscout/go/transport-metrics"
    metricsotel "gitlab.com/phpboyscout/go/transport-metrics/otel"
)

m, err := metrics.New(
    metrics.WithExemplarSource(metricsotel.ExemplarSource()),
)

Now every duration observation made on a context carrying a span gets a trace_id exemplar. metricsotel.ExemplarSource() reads only the span context. It starts no spans and needs no tracer provider, so it is safe whether or not tracing is configured; it yields nothing when the context carries no span.

It checks for the presence of a trace id, not for the sampling decision, so an unsampled span still produces an exemplar. That exemplar points at a trace your backend most likely never stored, so following it from a dashboard will dead-end. If that matters, wrap the source and drop unsampled spans yourself:

otelSource := metricsotel.ExemplarSource()

metrics.WithExemplarSource(func(ctx context.Context) (string, bool) {
    if !trace.SpanContextFromContext(ctx).IsSampled() {
        return "", false
    }
    return otelSource(ctx)
})

A custom source¶

Not on OpenTelemetry? Supply your own: anything that pulls a trace id from the request context:

metrics.WithExemplarSource(func(ctx context.Context) (string, bool) {
    if id := myTraceID(ctx); id != "" {
        return id, true
    }
    return "", false
})

Scraping exemplars¶

Exemplars ride on the OpenMetrics exposition format. The handler offers it, but only serves it when the client asks. Point a scraper that supports exemplars (Prometheus with --enable-feature=exemplar-storage, or an OTLP-scraping agent) at /metrics, and wire Grafana's exemplar-to-trace link on the trace_id label.

Why curl shows no exemplars¶

curl sends no Accept header, so it gets the Prometheus text format back, which has nowhere to put an exemplar. The samples are being recorded; the format is hiding them. Ask for OpenMetrics explicitly:

curl -s -H 'Accept: application/openmetrics-text; version=1.0.0' \
  localhost:8080/metrics | grep trace_id

http_request_duration_seconds_bucket{method="GET",route="GET /x",status="200",le="0.005"} 1 # {trace_id="0102030405060708090a0b0c0d0e0f10"} 2.2e-07 1.7856969705221822e+09

The exemplar is the part after the #: the trace id, the observed value, and the timestamp. If that comes back empty, the source is not yielding a trace id. Check there is a span on the context at the point the observation is made, not just somewhere upstream.

The bridge to OTel, not a conflict¶

Exemplars are the deliberate meeting point between this Prometheus (pull) module and the OTel (push) go/observability modules: your metrics stay in Prometheus, your traces stay in your tracing backend, and an exemplar stitches a sample to a trace across the two. It complements the OTel metrics story rather than competing with it.