Skip to content

Catch a memory leak with /metrics and pprof

This is the job the module was built for: a long-running Go process whose memory climbs, and no monitoring stack to find out why. You'll build a service that leaks on purpose, watch the leak on /metrics, then use pprof to name the line of code responsible. About fifteen minutes.

You need a Go toolchain of at least 1.26.5, curl, and go tool pprof (it ships with Go). Nothing else — no Prometheus server, no collector, no agent.

Build a service that leaks

The bug here is deliberate and obvious, so that the tools have something unambiguous to find. Every request appends a megabyte to a package-level slice that is never trimmed, so nothing is ever eligible for collection.

Save as main.go:

package main

import (
    "log"
    "net/http"

    metrics "gitlab.com/phpboyscout/go/transport-metrics"
)

// leaked keeps every chunk alive — the bug this tutorial hunts.
var leaked [][]byte

func main() {
    mux := http.NewServeMux()

    mux.HandleFunc("GET /work", func(w http.ResponseWriter, _ *http.Request) {
        leaked = append(leaked, make([]byte, 1<<20)) // 1 MiB, never released
        _, _ = w.Write([]byte("ok\n"))
    })

    if err := metrics.Register(mux, metrics.WithPprof()); err != nil {
        log.Fatal(err)
    }

    log.Println("listening on 127.0.0.1:8080")
    log.Fatal(http.ListenAndServe("127.0.0.1:8080", mux))
}

WithPprof() is what mounts the profiling handlers under /debug/pprof/. It is off unless you ask for it, because those handlers expose considerably more about the process than a counter does.

Bind to loopback before enabling pprof

The listener above is 127.0.0.1:8080, not :8080, and that matters more here than it did for plain metrics. A pprof endpoint will dump your heap — including whatever data happens to be in it — and the CPU and trace profiles hold the process while they run. Binding to loopback keeps it reachable from this machine only.

Run it:

go run .

WARN metrics endpoint mounted without guard middleware; ensure it is bound to loopback or fronted by auth path=/metrics
listening on 127.0.0.1:8080

The warning fires because no middleware was supplied; the loopback bind is the guard in this case. On anything shared, pass auth middleware instead — see Guard the endpoint.

Record a baseline

Take the numbers before you apply load, so you have something to compare against:

curl -s localhost:8080/metrics | grep -E '^(go_goroutines|go_memstats_heap_alloc_bytes|process_resident_memory_bytes) '

go_goroutines 6
go_memstats_heap_alloc_bytes 396432
process_resident_memory_bytes 1.4286848e+07

Three numbers worth knowing apart:

  • go_memstats_heap_alloc_bytes — live heap objects. This is the one a leak moves.
  • process_resident_memory_bytes — RSS as the OS sees it. It lags the heap, because the Go runtime does not return freed pages to the OS immediately.
  • go_goroutines — goroutine count. A goroutine leak shows up here and leaves the heap comparatively flat, which is how you tell the two apart.

Apply load and watch the heap climb

for i in $(seq 1 50); do curl -s localhost:8080/work > /dev/null; done

Then take the same reading again:

curl -s localhost:8080/metrics | grep -E '^(go_goroutines|go_memstats_heap_alloc_bytes|process_resident_memory_bytes) '

go_goroutines 6
go_memstats_heap_alloc_bytes 5.2991024e+07
process_resident_memory_bytes 1.6515072e+07

The heap has gone from roughly 400 KB to 53 MB and stays there — fifty requests, fifty megabytes retained. The goroutine count has not moved, so this is a memory leak, not a goroutine leak. RSS has barely risen yet; that is the runtime holding on to pages it may reuse, and it is why RSS alone is a poor leak signal over short windows.

Repeat the loop and read again if you want to be certain. A leak keeps climbing in step with the work; a normal working set rises and then plateaus.

Name the line of code with pprof

The endpoint tells you memory is being retained. pprof tells you by whom:

go tool pprof -top -nodecount=5 http://localhost:8080/debug/pprof/heap

Fetching profile over HTTP from http://localhost:8080/debug/pprof/heap
Type: inuse_space
Showing nodes accounting for 21671.22kB, 100% of 21671.22kB total
Showing top 5 nodes out of 23
      flat  flat%   sum%        cum   cum%
20132.66kB 92.90% 92.90% 20132.66kB 92.90%  main.main.func1
    1026kB  4.73% 97.63%     1026kB  4.73%  runtime.mallocgc
  512.56kB  2.37%   100%   512.56kB  2.37%  github.com/gogo/protobuf/proto.RegisterExtension
         0     0%   100%   512.56kB  2.37%  github.com/gogo/protobuf/gogoproto.init.0
         0     0%   100% 20132.66kB 92.90%  net/http.(*ServeMux).ServeHTTP

main.main.func1 is the /work handler, holding 93% of live heap. That is the answer.

The total pprof reports is smaller than the heap figure on /metrics — heap profiling samples roughly one allocation per 512 KB rather than recording all of them, so treat the proportions as reliable and the absolute bytes as an estimate. go_memstats_heap_alloc_bytes is the number to trust for how much; pprof is the tool for who.

For an interactive view of the same profile in a browser:

go tool pprof -http=: http://localhost:8080/debug/pprof/heap

If the process is under a rogue goroutine instead, swap the profile:

go tool pprof http://localhost:8080/debug/pprof/goroutine

Confirm the fix

Change the handler so the chunk goes out of scope:

mux.HandleFunc("GET /work", func(w http.ResponseWriter, _ *http.Request) {
    buf := make([]byte, 1<<20)
    _ = buf
    _, _ = w.Write([]byte("ok\n"))
})

Restart, run the same fifty requests, and read go_memstats_heap_alloc_bytes again:

go_memstats_heap_alloc_bytes 1.610432e+06

1.6 MB rather than 53 MB — a working set that reflects what is in flight, not a total that tracks the request count. Your exact figure will differ depending on when the collector last ran; what matters is that it no longer climbs in step with the load.

That comparison — baseline, load, re-read — is the whole loop, and it works the same way on a real service under real traffic.

Keep this out of production unguarded

Everything above ran against an endpoint reachable only from the host. In a deployed service, pprof needs a deliberate decision:

  • Pass your auth middleware with WithMiddleware, which wraps the pprof handlers as well as /metrics. See Guard the endpoint.
  • Or run the profiling endpoint on a separate loopback-bound listener with the standalone metrics server, which binds 127.0.0.1 by default.

Where to go next