Skip to content

Catch a memory leak with /metrics and pprof

This is the job the module was built for: a long-running Go process whose memory climbs, and no monitoring stack to find out why. You'll build a service that leaks on purpose, watch the leak on /metrics, then use pprof to name the line of code responsible. About fifteen minutes.

You need a Go toolchain that satisfies the go directive in this module's go.mod, curl, and go tool pprof (it ships with Go). Nothing else: no Prometheus server, no collector, no agent.

Build a service that leaks

The bug here is deliberate and obvious, so that the tools have something unambiguous to find. Every request appends a megabyte to a package-level slice that is never trimmed, so nothing is ever eligible for collection.

Save as main.go:

package main

import (
    "log"
    "net/http"

    metrics "gitlab.com/phpboyscout/go/transport-metrics"
)

// leaked keeps every chunk alive. This is the bug the tutorial hunts.
var leaked [][]byte

func main() {
    mux := http.NewServeMux()

    mux.HandleFunc("GET /work", func(w http.ResponseWriter, _ *http.Request) {
        leaked = append(leaked, make([]byte, 1<<20)) // 1 MiB, never released
        _, _ = w.Write([]byte("ok\n"))
    })

    if err := metrics.Register(mux, metrics.WithPprof()); err != nil {
        log.Fatal(err)
    }

    log.Println("listening on 127.0.0.1:8080")
    log.Fatal(http.ListenAndServe("127.0.0.1:8080", mux))
}

WithPprof() is what mounts the profiling handlers under /debug/pprof/. It is off unless you ask for it, because those handlers expose considerably more about the process than a counter does.

Bind to loopback before enabling pprof

The listener above is 127.0.0.1:8080, not :8080, and that matters more here than it did for plain metrics. A pprof endpoint will dump your heap, including whatever data happens to be in it, and the CPU and trace profiles hold the process while they run. Binding to loopback keeps it reachable from this machine only.

Run it:

go run .

WARN metrics endpoint mounted without guard middleware; ensure it is bound to loopback or fronted by auth path=/metrics
listening on 127.0.0.1:8080

The warning fires because no middleware was supplied; the loopback bind is the guard in this case. On anything shared, pass auth middleware instead. See Guard the endpoint.

Record a baseline

Take the numbers before you apply load, so you have something to compare against:

curl -s localhost:8080/metrics | grep -E '^(go_goroutines|go_memstats_heap_alloc_bytes|process_resident_memory_bytes) '

go_goroutines 6
go_memstats_heap_alloc_bytes 396432
process_resident_memory_bytes 1.4286848e+07

Three numbers worth knowing apart:

  • go_memstats_heap_alloc_bytes: live heap objects. This is the one a leak moves.
  • process_resident_memory_bytes: RSS as the OS sees it. It lags the heap, because the Go runtime does not return freed pages to the OS immediately.
  • go_goroutines: goroutine count. A goroutine leak shows up here and leaves the heap comparatively flat, which is how you tell the two apart.

Apply load and watch the heap climb

for i in $(seq 1 50); do curl -s localhost:8080/work > /dev/null; done

Then take the same reading again:

curl -s localhost:8080/metrics | grep -E '^(go_goroutines|go_memstats_heap_alloc_bytes|process_resident_memory_bytes) '

go_goroutines 6
go_memstats_heap_alloc_bytes 5.2991024e+07
process_resident_memory_bytes 1.6515072e+07

The heap has gone from roughly 400 KB to 53 MB and stays there: fifty requests, fifty megabytes retained. The goroutine count has not moved, so this is a memory leak, not a goroutine leak. RSS has barely risen yet; that is the runtime holding on to pages it may reuse, and it is why RSS alone is a poor leak signal over short windows.

Repeat the loop and read again if you want to be certain. A leak keeps climbing in step with the work; a normal working set rises and then plateaus.

Name the line of code with pprof

The endpoint tells you memory is being retained. pprof tells you by whom:

go tool pprof -top -nodecount=5 http://localhost:8080/debug/pprof/heap

Fetching profile over HTTP from http://localhost:8080/debug/pprof/heap
Type: inuse_space
Showing nodes accounting for 21671.22kB, 100% of 21671.22kB total
Showing top 5 nodes out of 23
      flat  flat%   sum%        cum   cum%
20132.66kB 92.90% 92.90% 20132.66kB 92.90%  main.main.func1
    1026kB  4.73% 97.63%     1026kB  4.73%  runtime.mallocgc
  512.56kB  2.37%   100%   512.56kB  2.37%  github.com/gogo/protobuf/proto.RegisterExtension
         0     0%   100%   512.56kB  2.37%  github.com/gogo/protobuf/gogoproto.init.0
         0     0%   100% 20132.66kB 92.90%  net/http.(*ServeMux).ServeHTTP

main.main.func1 is the /work handler, holding 93% of live heap. That is the answer.

The total pprof reports is smaller than the heap figure on /metrics because heap profiling samples roughly one allocation per 512 KB rather than recording all of them, so treat the proportions as reliable and the absolute bytes as an estimate. go_memstats_heap_alloc_bytes is the number to trust for how much; pprof is the tool for who.

For an interactive view of the same profile in a browser:

go tool pprof -http=: http://localhost:8080/debug/pprof/heap

If the process is under a rogue goroutine instead, swap the profile:

go tool pprof http://localhost:8080/debug/pprof/goroutine

Confirm the fix

Change the handler so the chunk goes out of scope:

mux.HandleFunc("GET /work", func(w http.ResponseWriter, _ *http.Request) {
    buf := make([]byte, 1<<20)
    _ = buf
    _, _ = w.Write([]byte("ok\n"))
})

Restart, run the same fifty requests, and read go_memstats_heap_alloc_bytes again:

go_memstats_heap_alloc_bytes 1.610432e+06

1.6 MB rather than 53 MB, a working set that reflects what is in flight, not a total that tracks the request count. Your exact figure will differ depending on when the collector last ran; what matters is that it no longer climbs in step with the load.

That comparison (baseline, load, re-read) is the whole loop, and it works the same way on a real service under real traffic.

Keep this out of production unguarded

Everything above ran against an endpoint reachable only from the host. In a deployed service, pprof needs a deliberate decision:

  • Pass your auth middleware with WithMiddleware, which wraps the pprof handlers as well as /metrics. See Guard the endpoint.
  • Or run the profiling endpoint on a separate loopback-bound listener with the standalone metrics server, which binds 127.0.0.1 by default.

Where to go next