0

I wanted to load a sample library (i.e. user32) and then using one exported function of that library like messageboxw to show a message to the user. My programs works fine, it shows the message but when I click on a button to close the program, it shows the following message:

Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention.

My source code:

#include <Windows.h>
#include <iostream>
#include <functional>

typedef int(*MsgBOX)
(
    HWND    hWnd,
    LPCWSTR lpText,
    LPCWSTR lpCaption,
    UINT    uType
);

int main(int argc, char* argv[])
{
    HINSTANCE handle_user32_dll = LoadLibrary(TEXT("User32.dll"));
    std::function<int(HWND, LPCWSTR, LPCWSTR, UINT)> MsgBoxInstance;
    
    if(!handle_user32_dll)
    {
        std::cout << "Dll isn't loaded successfuly." << std::endl;
    }
    else
    {
        MsgBoxInstance = reinterpret_cast<MsgBOX>(GetProcAddress(handle_user32_dll, "MessageBoxW"));
        if (!MsgBoxInstance)
        {
            std::cout << "Function didn't resolved.";
        }
        else
        {
            MsgBoxInstance(NULL, L"Resource not available\nDo you want to try again?", L"Account Details", MB_ICONWARNING | MB_CANCELTRYCONTINUE | MB_DEFBUTTON2);
        }
    }

    FreeLibrary(handle_user32_dll);

    return 0;
}

Where I did mistake and how can I fix this issue?

Timberwolf
  • 81
  • 7
  • Is there a reason you don't just link to user32 and then use `MessageBoxW` from the windows headers? If this really is the path you want to go down then you need to make sure your typedef matches exactly the function declaration in the headers including the calling convention – Alan Birtles Dec 28 '20 at 08:17
  • You want to call DLL functions, you need to know [stdcall and cdecl](https://stackoverflow.com/questions/3404372/stdcall-and-cdecl). If you got this code from a tutorial, it should have mentioned these calling conventions in bold, loud terms. – PaulMcKenzie Dec 28 '20 at 08:18
  • 1
    Try `typedef int WINAPI (*MsgBOX) ...` instead. That said, dynamically loading `user32` is unusual and seldom justified. – dxiv Dec 28 '20 at 08:20
  • [More about WINAPI](https://stackoverflow.com/questions/2348442/what-does-winapi-in-main-function-mean). It is just a macro denoting `__stdcall`. – PaulMcKenzie Dec 28 '20 at 08:22
  • @pau Calling conventions matter on every single function call. This is unrelated to DLL's. – IInspectable Dec 28 '20 at 08:25
  • @IInspectable That's true, but it becomes even more prominent when calling DLL functions. You could write a C++ program and never hear of the term "calling convention". If you're communicating with exported DLL functions, you better know what calling conventions are. – PaulMcKenzie Dec 28 '20 at 08:27

1 Answers1

1

Based on comment, I solved the problem with changing signature of the typedef.

Corrected source code:

#include <Windows.h>
#include <iostream>
#include <functional>

typedef WINUSERAPI int(WINAPI *MsgBOX)
(
    _In_opt_ HWND hWnd,
    _In_opt_ LPCWSTR lpText,
    _In_opt_ LPCWSTR lpCaption,
    _In_ UINT uType
);

int main(int argc, char* argv[])
{
    HINSTANCE handle_user32_dll = LoadLibrary(TEXT("User32.dll"));
    std::function<int(HWND, LPCWSTR, LPCWSTR, UINT)> MsgBoxInstance;

    if(!handle_user32_dll)
    {
        std::cout << "Dll isn't loaded successfuly." << std::endl;
    }
    else
    {
        MsgBoxInstance = reinterpret_cast<MsgBOX>(GetProcAddress(handle_user32_dll, "MessageBoxW"));

        if (!MsgBoxInstance)
        {
            std::cout << "Function didn't resolved.";
        }
        else
        {
            MsgBoxInstance(NULL, TEXT("Resource not available\nDo you want to try again?"), TEXT("Account Details"), MB_ICONWARNING | MB_CANCELTRYCONTINUE | MB_DEFBUTTON2);
        }
    }

    FreeLibrary(handle_user32_dll);

    return 0;
}
Timberwolf
  • 81
  • 7