You use the file descriptor when you have an open file that you want to map into memory. In this case, you're creating an anonymous map (one not backed by a file) so the file descriptor isn't needed. Some implementations ignore fd for anonymous maps, some require it to be -1.
The second question is a syntax error (probably a typo). It probably should be something like:
heapbase = (char*)(((uintptr_t)mem+HEADER_SIZE+OBJECT_GRAIN-1)
    &~(OBJECT_GRAIN-1)) - HEADER_SIZE;
In that case, OBJECT_GRAIN will be a power of two and it's a way to get alignment to that power. For example, if it were 8, then ~(OBJECT_GRAIN-1) would be ~7 (~00...001112, which is ~11...110002) which, when ANDed with a value, could be used to force that value to the multiple-of-8 less than or equal to it.
In fact, it's definitely a transcription error somewhere (not necessarily you) because, when I download the JamVM from here and look in src/alloc.c, I get:
void initialiseAlloc(InitArgs *args) {
    char *mem = (char*)mmap(0, args->max_heap, PROT_READ|PROT_WRITE,
                                               MAP_PRIVATE|MAP_ANON, -1, 0);
    :
    << a couple of irrelevant lines >>
    :    
    /* Align heapbase so that start of heap + HEADER_SIZE is object aligned */
    heapbase = (char*)(((uintptr_t)mem+HEADER_SIZE+OBJECT_GRAIN-1)&
               ~(OBJECT_GRAIN-1))-HEADER_SIZE;
(note that your version is also missing the - immediately before HEADER_SIZE, something else that points to transcription problems).