How to track memory usage accurately?

Question

I am trying to build a small tool that will allow me to run a program and track memory usage through Go. I am using r.exec = exec.Command(r.Command, r.CommandArgs...) to run the command, and runtime.MemStats to track memory usage (in a separate go routine):

func monitorRuntime() {
    m := &runtime.MemStats{}
    f, err := os.Create(fmt.Sprintf("mmem_%s.csv", getFileTimeStamp()))
    if err != nil {
        panic(err)
    }
    f.WriteString("Time;Allocated;Total Allocated; System Memory;Num Gc;Heap Allocated;Heap System;Heap Objects;Heap Released;\n")
    for {
        runtime.ReadMemStats(m)
        f.WriteString(fmt.Sprintf("%s;%d;%d;%d;%d;%d;%d;%d;%d;\n", getTimeStamp(), m.Alloc, m.TotalAlloc, m.Sys, m.NumGC, m.HeapAlloc, m.HeapSys, m.HeapObjects, m.HeapReleased))
        time.Sleep(5 * time.Second)
    }
}

When I tested my code with simple program that just sits there (for about 12 hours), I noticed that Go is constantly allocating more memory: System Memory Heap Allocation

I did a few more tests such as running the monitorRuntime() function without any other code, or using pprof, such as:

package main

import (
    "net/http"
    _ "net/http/pprof"
)

func main() {
    http.ListenAndServe(":8080", nil)
}

Yet I still noticed that memory allocation keeps going up just like in the graphs.

How can I accurately track memory usage of the program I want to run through Go? I know one way, which I used in the past, is to use /proc/$PID/statm, but that file doesn't exist in every operating system (such as MacOS or Windows)

Note that every step of that for loop is going to allocate some memory (mostly because of WriteString, iirc), and that memory will persist until garbage collection. Saw the same thing in my own loop, which allocated a hundred or so bytes per loop iteration until garbage collection. It's not a leak, it's just normal allocation. — Kaedys, Jan 31 '17 at 16:00
@VaD: ReadMemStats is as accurate as you can get, as that shows all allocations made by the Go runtime. Maybe track the output of `gctrace` to see how memory is being collected on your system. — JimB, Jan 31 '17 at 16:37
@Kaedys I did some research before asking that question, and I arrived to the same conclusion. Even though it occurs in simpler programs such as writing the output to stdout instead of a file, or using pprof. But I still cannot figure out how I can get accurate memory usage tracking of the command I run through Go (using `exec.Command`), I want to be able to tell if that command is leaking memory. — VaD, Jan 31 '17 at 16:39

score 1 · Answer 1 · answered Sep 12 '22 at 23:52

There isn't a way in standard Go to get the memory usage of a program called from exec.Command. runtime.ReadMemStats only returns memory tracked by the go runtime (which, in this case, is only the file handling and sprintf).

Your best bet would be to execute platform specific commands to get memory usage.

score 0 · Answer 2 · answered Oct 31 '18 at 15:40

0

On Linux (RedHat) the following will show memory usage: ps -No pid,comm,size,vsize,args:90

answered Oct 31 '18 at 15:40

Bill Zelenko

2,606
1
17
26

How to track memory usage accurately?

2 Answers2