Hi all and sorry for the cryptic title.
I have an executable which loads a DLL using WinAPI LoadLibrary/GetProcAddress. The program calls an exported function with C linkage with a custom class as parameter (as a reference but I will come back later on this). The exported function is defined as follow (GGen will be defined just after):
// LIB_API is __declspec(dllexport) in DLL context and
// __declspec(dllimport) in executable context
LIB_API void dbgt_generate(IRFile& cr) {
GGen::generate(cr);
}
My custom class is defined as follow and is header-only:
class IRFile {
public:
std::string identifier;
using callback_t = std::string(*)(IRFile*, void*);
static callback_t _callback;
virtual std::string buildFile(void* genData) {
assert(IRFile::_callback);
return IRFile::_callback(this, genData);
}
};
inline IRFile::callback_t IRFile::_callback = nullptr;
The exported function calls a singleton class by the way of a static member (the generate method):
static std::string cbFile(IRFile* _this, void* r) {
return "irfile";
}
class GGen {
public:
GGen() noexcept {
IRFile::_callback = cbFile;
}
static void generate(IRFile& cr) {
static GGen ggen;
assert(IRFile::_callback);
auto data = cr.buildFile(nullptr);
assert(!data.empty());
}
};
The executable simply initializes an IRFile instance on the stack and calls the exported function. My problem is that GGen::generate works well and assert pass without any problem but when buildFile is called during GGen::generate, the assert in it does not pass. At the precise cr.buildFile call moment, IRFile::_callback appears to be NULL.
After investigations, it appears that IRFile::_callback "has another address when in the virtual function compared to before" (I know this sound weird). I have figured out that removing virtual does fix the problem but in my complete program, I need it even if not used here. Another point is that when I pass the IRFile instance to the exported function as a copy (pass-by-value) the problem is also gone. But I don't understand why this last point fixes the problem as even if memory sharing was problematic, IRFile::_callback is initialized inside and by the DLL itself... I have added the identifier field to ensure correct memory is used and it is the case (in other words, this is correct).
Is it stack corruption? Why does removing virtual changes anything as, in my knowledges, static has nothing to do with vtables? Is this the effect of an undefined behavior with coincidences?
Please don't mind ugly C++ or naming conventions, I had to rewrite parts for testing to simplify.
[Edit 1] Clarified that the problem occurs after IRFile::_callback is initialized by GGen::generate and indirectly GGen ctor.
[Edit 2] As suggested in comments, I have tried to inline IRFile::_callback and export it as DLL symbol without success. I have also tried to put IRFile in a separate project (.lib and .dll tested) inlined or not, defined in a .cpp or not. I have also tried to encapsulate calls to _callback with setters/getters without success event when in a separate project (.lib/.dll).
[Edit 3] Added inline specifier to the _callback definition in header file to avoid multiple definition of _callback according to C++17:
