What exactly happens when I use the return value of a void function (by casting a function pointer)?

Question

When I run the following program, it always prints "yes". However when I change SOME_CONSTANT to -2 it always prints "no". Why is that? I am using visual studio 2019 compiler with optimizations disabled.

#define SOME_CONSTANT -3

void func() {
    static int i = 2;
    int j = SOME_CONSTANT;
    i += j;
}

void main() {
    if (((bool(*)())func)()) {
        printf("yes\n");
    }
    else {
        printf("no\n");
    }
}

EDIT: Here is the output assembly of func (IDA Pro 7.2):

sub     rsp, 18h
mov     [rsp+18h+var_18], 0FFFFFFFEh
mov     eax, [rsp+18h+var_18]
mov     ecx, cs:i
add     ecx, eax
mov     eax, ecx
mov     cs:i, eax
add     rsp, 18h
retn

Here is the first part of main:

sub     rsp, 628h
mov     rax, cs:__security_cookie
xor     rax, rsp
mov     [rsp+628h+var_18], rax
call    ?func@@YAXXZ    ; func(void)
test    eax, eax
jz      short loc_1400012B0

Here is main decompiled:

int __cdecl main(int argc, const char **argv, const char **envp)
{
  int v3; // eax

  func();
  if ( v3 )
    printf("yes\n");
  else
    printf("no\n");
  return 0;
}

Answer 1

((bool(*)())func)()

This expression takes a pointer to func, casts the pointer to a different type of function, then invokes it. Invoking a function through a pointer-to-function whose function signature does not match the original function is undefined behavior which means that anything at all might happen. From the moment this function call happens, the behavior of the program cannot be reasoned about. You cannot predict what will happen with any certainty. Behavior might be different on different optimization levels, different compilers, different versions of the same compiler, or when targeting different architectures.

This is simply because the compiler is allowed to assume that you won't do this. When the compiler's assumptions and reality come into conflict, the result is a vacuum into which the compiler can insert whatever it likes.

The simple answer to your question "why is that?" is, quite simply: because it can. But tomorrow it might do something else.

Answer 2

What apparently happened is:

mov     ecx, cs:i
add     ecx, eax
mov     eax, ecx   ; <- final value of i is stored in eax
mov     cs:i, eax  ; and then also stored in i itself

Different registers could have been used, it just happened to work this way. There is nothing about the code that forces eax to be chosen. That mov eax, ecx is really redundant, ecx could have been stored straight to i. But it happened to work this way.

And in main:

call    ?func@@YAXXZ    ; func(void)
test    eax, eax
jz      short loc_1400012B0

rax (or part of it, like eax or al) is used for the return value for integer-ish types (such as booleans) in the WIN64 ABI, so that makes sense. That means the final value of i happens to be used as the return value, by accident.

Answer 3

I always get printed out no, so it must be dependent from compiler to compiler, hence the best answer is UB (Undefined Behavior).