How can I know how much memory an STL object takes?

Question

I need to gather statistics about memory usage in my program.

My code is mostly written using STL.

Is there any way of learning how much memory is being consumed by an STL object?

For example,

string s1 = "hello";
string s2 = "hellohellohellohellohellohellohellohellohellohellohellohellohello";

How much memory is consumed by s1 and s2? Obviously, sizeof(string)+s1.length() is not quite accurate.

Answer 1

If you're willing to be slightly intrusive, you can make a custom allocator to track all heap usage by container type, or by type allocated. This is quite intrusive, but quite accurate as well, for determining memory usage. This does not track how much memory the heap itself takes though, as that is highly OS-dependent.

template<class TrackType> 
size_t* mem_used() {static size_t s = 0; return &s;}

template<class T, class TrackType, class BaseAllocator = std::allocator<T> > 
class TrackerAllocator : public BaseAllocator {
public:
    typedef typename BaseAllocator::pointer pointer;
    typedef typename BaseAllocator::size_type size_type;

    TrackerAllocator() throw() : BaseAllocator() {}
    TrackerAllocator(const TrackerAllocator& b) throw() : BaseAllocator(b) {}
    TrackerAllocator(TrackerAllocator&& b) throw() : BaseAllocator(b) {}
    template <class U> TrackerAllocator(const typename TrackerAllocator::template rebind<U>::other& b) throw() : BaseAllocator(b) {}
    ~TrackerAllocator() {}

    template<class U> struct rebind {
        typedef TrackerAllocator<U, TrackType, typename BaseAllocator::template rebind<U>::other> other;
    };

    pointer allocate(size_type n) {
        pointer r = BaseAllocator::allocate(n);
        *mem_used<TrackType>() += n;
        return r;
    }
    pointer allocate(size_type n, pointer h) {
        pointer r = BaseAllocator::allocate(n, h);
        *mem_used<TrackType>() += n;
        return r;
    }
    void deallocate(pointer p, size_type n) throw() {
        BaseAllocator::deallocate(p, n);
        *mem_used<TrackType>() -= n;
    }
};

And usage is:

typedef std::basic_string<char, 
                          std::char_traits<char>,
                          TrackerAllocator<char, std::string> > trackstring;
typedef std::vector<int, 
                    TrackerAllocator<int, std::vector<int> > > trackvector;
//                                        ^              ^
//                                        This identifies which memory to track
//                                        it can be any type, related or no.
//                                        All with the same type will be tracked togeather
int main() {
    trackstring mystring1("HELLO WORLD");
    std::cout << *mem_used<std::string>() << '\n'; //display memory usage of all strings

    trackstring mystring2("MUCH LONGER STRING THAT DEFINITELY GETS HEAP ALLOCATED!");
    std::cout << *mem_used<std::string>() << '\n'; //display memory usage of all strings

    trackvector myvec(mystring1.begin(), mystring1.end());
    std::cout << *mem_used<std::vector<int> >() << '\n'; //display memory usage of all vector<int>
    //                     ^              ^
    //                     This identifies which memory type from above to look up.
    return 0;
}

Windows results:

0 //this is zero, because the string did not allocate heap space.
64
11

http://ideone.com/lr4I8 (GCC) results:

24
92
11

Answer 2

Since this is completely implementation details, you cannot determine this with 100% accuracy.

However, as you said, you want to put some code to statistics the memory usage in your program, then you can do that with some accuracy.

I believe, for std::string, the size of memory taken by string objects, will be almost equal to:

size_t statisticalSizeStr = sizeof(string)+ s.capacity() * sizeof(char);

And similarly, for std::vector

size_t statisticalSizeVec = sizeof(std::vector<T>)+ ....;

You can model your statistical estimation on such informations. For vector, you can also consider the size of T in the same way which will fill the .... in the above equation. For example, if T is std::string, then:

size_t vecSize = sizeof(std::vector<std::string>);
size_t statisticalSizeVec = vecSize  + v.capacity() * statisticalSizeStr;

And if T is int, then

size_t statisticalSizeVec=sizeof(std::vector<int>)+v.capacity()*sizeof(int);

I hope, such analysis would help you to compute the size with as much accuracy as possible.

Answer 3

Just for fun, I made my best attempt to find the memory usage of the individual strings you gave. I concur with people who say this is basically impossible; my implementation is not good in many ways:

• #define private public gives away that I'm "doing it wrong" and depending on something which is not only not standardized but even for my STL implementation (gcc 4.6) might change by next release. You should never see this in production code.
• I think you're looking for something usable for any STL object, and I only implemented std::string. You'd have to do specific logic for each type as no general mechanism exists on my platform. And for containers, you'd have to recurse; there are no existing accurate counters for an O(1) get_allocated_size().
• My platform's strings are reference-counted; so if you add up sizes of a bunch of strings with my function you will overestimate any ones which share representations with each other.
• I also used malloc_usable_size to see the actual size of the area returned by malloc. First, this is specific to the allocator included with glibc. Second, it doesn't count malloc's book-keeping memory. Third, this allocation might cause some other allocation to take up more memory because it is introducing fragmentation in the virtual address space or some such. But it is more accurate than what the caller asked for as malloc tends to round things up.

In practice, I'd suggest make the sort of approximation Nawaz mentioned and validating it empirically with whole-process measurements as other people have mentioned.

With that said, here we go:

$ cat > sizetest.cc <<EOF
#define private public  // eww...

#include <malloc.h>

#include <iostream>
#include <string>

using namespace std;

// NON-PORTABLE! Totally dependent on gcc 4.6 / glibc (with the stock allocator)!    
size_t get_size(const string &s) {
  string::_Rep *rep = (string::_Rep*) s.data() - 1;
  return sizeof(string) + malloc_usable_size(rep);
}

int main(int argc, char **argv) {
  string s1 = "hello";
  string s2 = "hellohellohellohellohellohellohellohellohellohellohellohellohello";
  cout << "s1 size: " << get_size(s1) << endl;
  cout << "s2 size: " << get_size(s2) << endl;
  return 0;
}
EOF
$ g++ -Wall sizetest.cc -o sizetest
$ ./sizetest 
s1 size: 48
s2 size: 112

Answer 4

I don't think it would be possible to measure for an individual container.

However, to get an overall summary, you can use the mallinfo() function (on Linux).

If you want to know where are the biggest memory hogs in your program, you can run it under valgrind with its massif memory profiler, which will give you information like "15 MB were allocated from map<my_data>::...", so you can guess how large your structures are.

Answer 5

You can use the Psapi for this if you want general memory usage statistics.

    #ifdef WIN32
    #include <psapi.h>
    #elif __GNUC__
        #include <sys/time.h>
        #include <sys/resource.h>
    #endif

void MemoryUsage()
{
#ifdef WIN32
    PROCESS_MEMORY_COUNTERS pmc;
    if(GetProcessMemoryInfo(GetCurrentProcess(),&pmc,sizeof(pmc)))
    {
                // do something with the values you care about
        pmc.PagefileUsage;
        pmc.PeakPagefileUsage;
        pmc.PeakWorkingSetSize;
        pmc.QuotaNonPagedPoolUsage;
        pmc.QuotaPagedPoolUsage;
        pmc.QuotaPeakNonPagedPoolUsage;
        pmc.WorkingSetSize;
        }
#else
    // I'm not a penguin
#endif
}