Which exception can be generated when subtracting ESP or RSP register? (stack growing)

Question

I'm trying to understand how exactly memory pages for stack is allocated/assigned.

I wrote the following proof-of-concept C-code which obviously causes segmentation fault (on x86_64 Linux):

#include <string.h>

int main()
{
    char a;

    memset( (&a - 4444444), 0, 3333333 );

    return 0;
}

The following fragment of assembly code (AT&T syntax) is generated by gcc from above C-program:

subq    $16, %rsp
leaq    -1(%rbp), %rax
subq    $4444444, %rax
movl    $3333333, %edx
movl    $0, %esi
movq    %rax, %rdi
call    memset

If I add subq $5555555, %rsp manually before calling memset:

subq    $16, %rsp
leaq    -1(%rbp), %rax
subq    $4444444, %rax
movl    $3333333, %edx
movl    $0, %esi
movq    %rax, %rdi
subq    $5555555, %rsp /* added manually */
call    memset

Then segmentation fault disappears because virtual memory pages for stack was assigned after subtracting rsp register caused some hardware exception and assigned exception handler was called (of course, in kernel space).

I know that calling memset here will cause "minor page fault" exceptions. But it's a different story (i.e. allocating physical memory pages).

My question is: Which exception was generated when subq $5555555, %rsp is invoked? I suggest it would be "stack fault" exception but I did not find exact proof for it.

Answer 1

I figured it out. First of all, subtracting rsp register does nothing. Second, when we try to write to non-mapped stack area "minor page fault" exception handler is invoked in kernel space. Then this page fault handler checks whether it was legal write or non-legal. I think page fault handler compares with current stack pointer of the thread (in our case it's saved value rsp register). If address where process try to write is upper than current stack pointer then page fault handler expand process's virtual address space and maps this virtual page to physical page, otherwise handler sends SIGSEGV to the process.

I examined the following fragment by using GDB and /proc/[pid]/maps:

subq    $1500016, %rsp
movq    %fs:40, %rax
movq    %rax, -8(%rbp)
xorl    %eax, %eax
movb    $44, -1500016(%rbp)
movb    $55, -1100016(%rbp)
movb    $66, -600016(%rbp)

When subq $1500016, %rsp is invoked stack address range isn't changed. But when first write happens by movb $44, -1500016(%rbp), stack address range is expanded as I explained above.

Answer 2

There is no exception on that line.

However, memset's prologue code will cause an access violation when it tries to preserve registers by saving them to the stack, as the stack pointer is invalid.

In most environments, there is only a single guard page that triggers additional stack pages to be committed. In that case the access violation will not be handled by growing the stack, the program will simply crash.

In case your OS actually does handle the access violation caused during register save, it will commit all the intervening pages of stack and retry the operation (the PUSH instruction). Then those intervening pages will be successfully written by the loop inside memset.

Of course, if the subtraction causes RSP to point outside the address space reserved for stack growth, all bets are off. You could even cause the stack of some other thread to grow.