Unexpected MPI address offset

Question

In the code below, I expected addr[2] - addr[1] to be equal to 4x4=16 which is the answer given by block_count[1]*extent_int. Why addr[2] - addr[1] gives 20?

#include <mpi.h>
#include <iostream>

struct S
{
    int a;
    int nei[4];
    double point[4];
};

int main()
{
    MPI_Init(NULL, NULL);

    S s;

    int nblock = 3;
    // block count.
    int block_count[nblock] = {1, 4, 4};
    // extent.
    MPI_Aint lb, extent_int;
    MPI_Type_get_extent(MPI_INT, &lb, &extent_int);
    // offset.
    MPI_Aint addr[nblock];
    MPI_Get_address(&s.a, &addr[0]);
    MPI_Get_address(&s.nei[0], &addr[1]);
    MPI_Get_address(&s.point[0], &addr[2]);
    //
    std::cout << addr[1]-addr[0] << "  " << block_count[0]*extent_int << std::endl;
    std::cout << addr[2]-addr[1] << "  " << block_count[1]*extent_int << std::endl;

    MPI_Finalize();

    return 0;
}

Output:

4  4
20  16

This seems to be related to alignement. See http://stackoverflow.com/questions/11108328/double-alignment . In particular, [`On x86-64, -malign-double is enabled by default.`.](https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html) Here, the space between the beginning of the structure and the array of double is 24 bytes, a multiple of 8. — francis, Jan 13 '17 at 17:48

score 1 · Accepted Answer · edited May 23 '17 at 12:26

This has nothing to do with MPI. On architectures with flat address space (basically any modern OS) MPI_Get_address(&a, &b); is equivalent to b = (MPI_Aint)&a; You simply observe (as pointed out by @francis) your C++ compiler inserting padding in order to satisfy the default alignment rules for your platform. The rest is described in the answers to this question (the links provided by @francis).

There is a relatively simple rule to prevent misalignment and the associated increase in size of such structures: put (if possible) the biggest types at the beginning:

struct S
{
   double point[4];
   int a;
   int nei[4];
};

Since the size of int divides the size of double, the ints will be aligned and no padding will be added in that case. The same applies to the local variables in your code, and some compilers do rearrange them as part of the code optimisation process.

Unexpected MPI address offset

1 Answers1