Link metrics to traces (exemplars)¶

An exemplar attaches a trace id to a single metric sample — not a label — so a latency spike on a Prometheus/Grafana dashboard links straight to the trace that produced it. You get the drill-down with zero cardinality cost, because the trace id never becomes a label.

transport-metrics records exemplars on its duration histograms (http_request_duration_seconds, grpc_server_handling_seconds) whenever an exemplar source is configured.

Enable it¶

The trace-id source is pluggable so the core needs no tracing dependency. Use the opt-in metrics/otel subpackage to read the current OpenTelemetry span:

import (
    metrics "gitlab.com/phpboyscout/go/transport-metrics"
    metricsotel "gitlab.com/phpboyscout/go/transport-metrics/otel"
)

m, err := metrics.New(
    metrics.WithExemplarSource(metricsotel.ExemplarSource()),
)

Now every HTTP/gRPC duration observation made on a context carrying a sampled span gets a trace_id exemplar. metricsotel.ExemplarSource() reads only the span context — it starts no spans and needs no tracer provider — so it is safe whether or not tracing is configured; it simply yields nothing when there is no span.

A custom source¶

Not on OpenTelemetry? Supply your own — anything that pulls a trace id from the request context:

metrics.WithExemplarSource(func(ctx context.Context) (string, bool) {
    if id := myTraceID(ctx); id != "" {
        return id, true
    }
    return "", false
})

Scraping exemplars¶

Exemplars ride on the OpenMetrics exposition format, which this module's handler negotiates automatically. Point a scraper that supports exemplars (Prometheus with --enable-feature=exemplar-storage, or an OTLP-scraping agent) at /metrics, and wire Grafana's exemplar-to-trace link on the trace_id label.

The bridge to OTel, not a conflict¶

Exemplars are the deliberate meeting point between this Prometheus (pull) module and the OTel (push) go/observability modules: your metrics stay in Prometheus, your traces stay in your tracing backend, and an exemplar stitches a sample to a trace across the two. It complements the OTel metrics story — it does not compete with it.