How to reduce the memory consumption of unordered_multiset?

Question

I used unordered_multiset in my code by the following two reasons,

Should be easy to find or look up the data.
Should supports to load duplicate values.

unordered_multiset are typically much faster than multisets & vector, both for insertion and for lookup, and sometimes even for deletion.

But the bad thing is,it takes too much of memory.

I have stored unsigned __int64 (8 bytes)values in the unordered_multiset and properly clear the values from the unordered_multiset. can you anyone explain,why its taking the memory & how to solve this memory consumption?

@Drek i got the memory consumption details from Task Manager. I find it using process id of my application. — Durai, Aug 04 '17 at 09:52
How many *distinct* items do you insert? How large is the memory footprint of your `unordered_multiset` once all values have been inserted? — Sergey Kalinichenko, Aug 04 '17 at 09:55
_"But the bad thing is,it takes too much of memory."_: we need numbers. — Richard Critten, Aug 04 '17 at 09:56
"from task manager" - not the most reliable way to get information about a single container. Unless you show code and the memory used as it goes along how can people help? — UKMonkey, Aug 04 '17 at 09:57
Before the values inserted into unordered_multiset my application takes 82568K After the values inserted into unordered_multiset my application takes 141986K I have 50 to 200 values(max) in the unordered_multiset. approx 30 distinct values. — Durai, Aug 04 '17 at 10:46
what else happened in your program, in the time between those two task-manager updates ? — Caleth, Aug 04 '17 at 10:48
@UKMonkey cool, i know the Task manager shows the overall memory details for the process. I checked the memory consumption by commenting insertion of Unordered_multiset. — Durai, Aug 04 '17 at 10:49
For an ex. if i insert 200 values (8 bytes per value) in the unordered_multiset 200*8 =1600 bytes can increase in heap memory . why this much of difference? — Durai, Aug 04 '17 at 10:53
You've potentially got different executable sizes wierding up your measurements — Caleth, Aug 04 '17 at 10:54

score 0 · Answer 1 · answered Aug 04 '17 at 11:10

You can get a much better measurement of how much space a std::container uses by providing it with a custom allocator that logs how much it is asked to allocate.

e.g.

std::size_t total_allocation = 0;

template< class T >
struct logging_allocator
{   
    using value_type = T;
    using pointer = T*;
    using const_pointer = const T*;
    using reference = T&;
    using const_reference = const T&;
    using size_type = std::size_t;
    using difference_type = std::ptrdiff_t;
    using propagate_on_container_move_assignment = std::true_type;
    template< class U > struct rebind { using other = logging_allocator<U>; };
    using is_always_equal = std::true_type;

    pointer address( reference x ) const { return base.address(x); }
    const_pointer address( const_reference x ) const{ return base.address(x); }

    pointer allocate( size_type n, const_pointer hint = nullptr ){ 
        total_allocation += n; 
        return base.allocate(n, hint); 
    }
    pointer allocate( size_type n, const void * hint){ 
        total_allocation += n; 
        return base.allocate(n, hint); 
    }
    pointer allocate( size_type n ){ 
        total_allocation += n; 
        return base.allocate(n, hint); 
    }

    void deallocate( T* p, size_type n ) {
        total_allocation -= n; 
        return base.deallocate(p, n);
    }

    size_type max_size() const {
        return base.max_size();
    }

    void construct( pointer p, const_reference val ) {
        total_allocation += sizeof(T); 
        return base.construct(p, val);
    }
    template< class U, class... Args >
    void construct( U* p, Args&&... args ) {
        total_allocation += sizeof(U); 
        return base.construct(p, args...);
    }

    void destroy( pointer p ) {
        total_allocation -= sizeof(T); 
        return base.destroy(p);
    }
    template< class U >
    void destroy( U* p ) {
        total_allocation -= sizeof(U); 
        return base.destroy(p);
    }

private:
    std::allocator<T> base;
}

score 0 · Answer 2 · answered Aug 06 '17 at 13:40

0

Caleth has a good suggestion, alternatively you can look at memory usage within processes

just before you insert into the multiset and after.

Most likely course of difference, a huge dll is loaded.

answered Aug 06 '17 at 13:40

Surt

15,501
3
23
39

How to reduce the memory consumption of unordered_multiset?

2 Answers2