Why does mstats and malloc_zone_statistics not show recovered memory after free?

Question

I'm trying to write some unit tests that check whether memory has been freed -- to check for memory leaks -- on OS X (10.9 Mavericks). I'm trying to use mstats() and malloc_zone_statistics() to discover the information I need. But it seems that they don't show memory being released (see example output below ... memory usage does not change after free() is called)

I suspect this has more to do with the heap management than with a problem with those functions. I think the heap is not releasing freed memory, perhaps so it can reuse it without the overhead of removing and adding blocks.

1. Is there some way to tell the heap to release freed blocks? To be more aggressive or turn off optimizations?
2. Am I just using mstats() or malloc_zone_statistics() incorrectly?

Update: solution found ... provided at the bottom ...

Here is the output from my test program:

=== Initial conditions ===
in use: 23584, allocated: 9437184, blocks: 320
SimpleLeaker(19583,0x7fff7b2a2310) malloc: total: 9437184, used: 23584, free: 9413600
=== Before allocation ===
in use: 23584, allocated: 9437184, blocks: 320
SimpleLeaker(19583,0x7fff7b2a2310) malloc: total: 9437184, used: 23584, free: 9413600
=== After malloc ===
in use: 33824, allocated: 9437184, blocks: 321
SimpleLeaker(19583,0x7fff7b2a2310) malloc: total: 9437184, used: 33824, free: 9403360
=== After free ===
in use: 33824, allocated: 9437184, blocks: 321
SimpleLeaker(19583,0x7fff7b2a2310) malloc: total: 9437184, used: 33824, free: 9403360

And here is the C code for the program:

    #include <stdio.h>
    #include <unistd.h>
    #include <stdlib.h>
    #include <pthread.h>
    #include <mach/mach.h>
    #include <mach/task.h>
    #include <malloc/malloc.h>
    #include <errno.h>


    /** heapStats()
     * print the output from the mstats() function: total heap bytes, 
     * used bytes, and free bytes.
     */
    void heapStats()
    {
      struct mstats ms = mstats();
      malloc_printf(
        "total: %d, used: %d, free: %d\n",
        ms.bytes_total,
        ms.bytes_used,
        ms.bytes_free);
    }

    /* heapInUse()
     * Gather the heap usage metrics from each zone, using 
     * malloc_zone_statistics().
     */
    void heapInUse(
      size_t * bytesInUse,
      size_t * blocksInUse,
      size_t * sizeAllocated)
    {
      *bytesInUse = 0;
      *blocksInUse = 0;
      *sizeAllocated = 0;
      unsigned int i;
      vm_address_t * zones;
      unsigned int count;
      kern_return_t rc =
      malloc_get_all_zones(mach_task_self(), 0, &zones, &count);
      if (0 != rc)
      {
        fprintf(stderr, "rc was %d\n", rc);
      }
      for (i = 0; i < count; ++i)
      {
        malloc_zone_t * zone = (malloc_zone_t*)zones[i];
        char const * name = malloc_get_zone_name(zone);
        if (NULL == name)
        {
          continue;
        }
        malloc_statistics_t stats;
        stats.blocks_in_use = 0;
        stats.size_in_use = 0;
        stats.max_size_in_use = 0;
        stats.size_allocated = 0;
        malloc_zone_statistics(zone, &stats);
        *bytesInUse += stats.size_in_use;
        *blocksInUse += stats.blocks_in_use;
        *sizeAllocated += stats.size_allocated;
      }
    }

    /** main()
     * entry point
     */
    int main(int argc, const char * argv[])
    {
      char * buff = (char *)0;
      size_t bytesInUse = 0;
      size_t blocksInUse = 0;
      size_t sizeAllocated = 0;
      printf("=== Initial conditions ===\n");
      heapInUse(&bytesInUse, &blocksInUse, &sizeAllocated);
      printf(
        "in use: %zu, allocated: %zu, blocks: %zu\n",
        bytesInUse, sizeAllocated, blocksInUse);
      heapStats();

      printf("=== Before allocation ===\n");
      heapInUse(&bytesInUse, &blocksInUse, &sizeAllocated);
      printf(
        "in use: %zu, allocated: %zu, blocks: %zu\n",
        bytesInUse, sizeAllocated, blocksInUse);
      heapStats();

      // Allocate the buffer
      //
      buff = (char *)malloc(10000);
      printf("=== After malloc ===\n");
      heapInUse(&bytesInUse, &blocksInUse, &sizeAllocated);
      printf(
        "in use: %zu, allocated: %zu, blocks: %zu\n",
        bytesInUse, sizeAllocated, blocksInUse);
      heapStats();

      // Free the buffer
      //
      if (NULL != buff)
      {
        free(buff);
        buff = NULL;
      }
      printf("=== After free ===\n");
      heapInUse(&bytesInUse, &blocksInUse, &sizeAllocated);
      printf(
        "in use: %zu, allocated: %zu, blocks: %zu\n",
        bytesInUse, sizeAllocated, blocksInUse);
      heapStats();

      // Get out
      //
      return 0;
    }

Solution: Thanks to the response from John Zwinck I took a closer look at malloc/malloc.h and found a method malloc_zone_pressure_relief() that I can use to force the heap to free unused bytes so that I can get accurate metrics.

So I added this method:

void freeAsMuchAsPossible()
{
  vm_address_t * zones;
  unsigned int count;
  unsigned int i;

  kern_return_t rc =
    malloc_get_all_zones(mach_task_self(), 0, &zones, &count);
  if (0 != rc)
  {
    fprintf(stderr, "rc was %d\n", rc);
  }
  for (i = 0; i < count; ++i)
  {
    malloc_zone_t * zone = (malloc_zone_t*)zones[i];
    char const * name = malloc_get_zone_name(zone);
    if (NULL == name)
    {
      continue;
    }
    malloc_zone_pressure_relief(zone, 0);
  }
}

and called it before every call to heapInUse(), like so:

printf("=== Before allocation ===\n");
freeAsMuchAsPossible();
heapInUse(&bytesInUse, &blocksInUse, &sizeAllocated);
printf(
  "in use: %zu, allocated: %zu, blocks: %zu\n",
  bytesInUse, sizeAllocated, blocksInUse);
heapStats();

And now I get expected and useful results, for example:

=== Initial conditions ===
in use: 23584, allocated: 9437184, blocks: 4294966976
SimpleLeaker(22142,0x7fff7b2a2310) malloc: total: 9437184, used: 23584, free: 9413600
=== Before allocation ===
in use: 23584, allocated: 9437184, blocks: 4294966976
SimpleLeaker(22142,0x7fff7b2a2310) malloc: total: 9437184, used: 23584, free: 9413600
=== After malloc ===
in use: 33824, allocated: 9437184, blocks: 4294966967
SimpleLeaker(22142,0x7fff7b2a2310) malloc: total: 9437184, used: 33824, free: 9403360
=== After free ===
in use: 23584, allocated: 9437184, blocks: 4294966966
SimpleLeaker(22142,0x7fff7b2a2310) malloc: total: 9437184, used: 23584, free: 9413600

With this technique I can write unit tests that check for memory leaks. Very nice.

Answer 1

On OS X, since 10.7, we can use malloc_zone_pressure_relief() to force the heap to release unused bytes. I've updated my question with the solution.

Thanks to the response from John Zwinck for prodding me into another close look at malloc.h where I found that method.

Answer 2

You're probably right: the allocator isn't releasing memory back to the OS every time you call free(). But you can change this using mallopt(). Try setting M_MMAP_THRESHOLD to 0. This will make every single allocation independent. You probably don't want this for production, but for testing it may help you.

Answer 3

I will suggest looking into the OSX leaks tool to find memory leaks in your program. An introductory article can be found here: leaks documentation at Mac Developer Library. You can do man leaks for more details. Also, just for your question 1, you can use libgmalloc. Do man libgmalloc to learn more. Also, these man pages contain links to other tools such as malloc_history that you may also find useful and/or faster for your purpose.