I have an old C Win16 application that is behaving badly when run in Windows 10 32bit. Unfortunately, I do not have the source code for the application so I cannot debug issues from there. The application was working normally in Windows 7 but in Windows 10, it starts by using NTVDM but for some reason the GUI has the following issues:
Issue 1. The application opens a popup dialog when it starts. Mouse interaction works so that I can drag the popup dialog around the application window but the popup itself has no focus and I cannot click any of the buttons or input text to the dialog. I can use still Alt+F to access File-menu in the top application bar while the popup is open.
Issue 2. The application can also be configured to open without the popup window. In this case the top menu-bar and status bars are displayed out of place and do not respond to any input from the keyboard.
In both cases, clicking on the main application window buttons does nothing and I need to use task manager to kill ntvdm.exe to close the application.
So far, I have tried different compatibility mode settings as well as trying to run the application with an alternative VDM (WineVDM). Compatibility mode does nothing and the application launches using WineVDM but has same layout issues that using NTVDM has. This seems to indicate that the problem is not in VDM but has more to do with legacy graphics rendering differences between Win7 and Win10.
What should I try next? Is there a way to identify which GUI framework was used?
Edit: I'm currently running this in a virtual machine "Win10 32bit" inside a 64bit Windows 10 machine and the display adapter is currently "Microsoft Hyper-V Video". Tried changing this to "Microsoft Basic Display Adapter" but that didn't help.
Edit2: Read somewhere that Win10 creators update broke GDI applications so tested with an earlier version of windows (1607). So the question actually changes to "how can I get GDI applications to run correctly after creators update".