4

Can I change the VM flag in the EFLAGS register? If so, how?

Note that I can't just use pushf, manipulate the value on the stack, and then popf. According to the Intel manual:

Volume 2, Chapter 4.3 PUSHF:

"When copying the entire EFLAGS register to the stack, the VM and RF flags (bits 16 and 17) are not copied; instead, the values for these flags are cleared in the EFLAGS image stored on the stack."

Volume 2, Chapter 4.3 POPF:

"When operating in protected, compatibility, or 64-bit mode at privilege level 0 (or in real-address mode, the equiv- alent to privilege level 0), all non-reserved flags in the EFLAGS register except RF1, VIP, VIF, and VM may be modi- fied. VIP, VIF and VM remain unaffected."

Michael Petch
  • 46,082
  • 8
  • 107
  • 198
Muricula
  • 1,174
  • 3
  • 9
  • 16
  • 1
    To set the VM flag you either need to use IRET or load it from a TSS though a task change. Both require starting off in ring 0 and ending up in ring 3. There's no reason to set the VM flag except as a part of an operating system's real-mode emulator. – Ross Ridge Aug 28 '16 at 21:48
  • 2
    Note that in long mode (implied by the `x86-64` tag), virtual 8086 mode is not supported (even for CPL=0), so the VM flag in EFLAGS can not be set (unless you switch from long mode to protected mode first, which is likely to be several different flavours of crazy for most practical purposes). – Brendan Aug 29 '16 at 02:19

1 Answers1

7

No.

Flags that can be changed by a user-mode program
Only 8 of the 12 least significant bits can be set by user applications (CF, PF, AF ZF, SF, TF, DF OF), of those 12 bits you cannot set the reserved bits, leaving only 9 and one of these (IF) is privileged.
Most flags can be changed using normal instructions, a user application only needs POPF to set the TF flag.
The higher flags are privileged or readonly and can only be changed by a process running in ring 0.

See: http://www.felixcloutier.com/x86/POPF:POPFD:POPFQ.html

All non-reserved bits except IF, IOPL, VIP, VIF, VM and RF can be modified; IF, IOPL, VIP, VIF, VM and all reserved bits are unaffected; RF is cleared.
If a POPF/POPFD instruction is executed with insufficient privilege, an exception does not occur but privileged bits do not change.

See: https://en.wikipedia.org/wiki/FLAGS_register

How do I change the VM flag and what is it anyway?
The VM flag has nothing to do with virtual machines, but shows whether the processor is running in 16-bit virtual 8086 mode.
In order to set the VM flag you need to enter Virtual 8086 mode.
See here for example code.
Many of the instructions needed are privileged and can only be executed in Ring 0.
Also you can only enter into real mode from 32 bit mode, not from 64 bit mode.

Exiting from virtual 8086 mode back into protected mode is an even more convoluted affair (scroll down a bit for working code).
Again you need to run in privileged VME mode to be able to do this.

Johan
  • 74,508
  • 24
  • 191
  • 319
  • As I understand it, vm86 mode *is* a proper virtual 16-bit machine under a protected-mode OS. It's just that it's it doesn't support the guest switching to 32-bit protected mode. It does let the guest use privileged instructions that affect the virtual environment. – Peter Cordes Aug 28 '16 at 23:14
  • Given the fact that the OP lists `x86-64` as a tag I think he's under the mistaken assumption that the VM flag would tell a program whether it was running under VMware or such. Real mode is quite a ways removed from that. Real mode can be seen as a VM of sorts I guess if you squint your eyes just right. – Johan Aug 28 '16 at 23:37
  • Yeah, I agree with your general point that it's totally unrelated to modern VMs that can virtualize the full capabilities of the CPU to run "normal" OSes as guests. But vm86 mode does virtualize real mode, allowing 16-bit code to execute at native speed (except for hardware access / system calls). I'm not sure what you mean with your last sentence, though. Actual real mode (not VM86 mode) isn't a VM at all. VM86 mode is *like* real mode, but I wouldn't say "aka real mode", because Real Mode is a separate thing, e.g. for bootloaders. – Peter Cordes Aug 28 '16 at 23:48
  • I was lead astray because the VMXON instruction has undefined behavior when the VM flag is set. I thought I would have to clear it before entering VMX mode, but since I'm running in long mode this is probably not my problem. – Muricula Aug 29 '16 at 18:36
  • Entering v8086 mode (EFLAGS.VM=1, CR0.PE=1) and real mode are two different things. When in real mode EFLAGS.VM = 0 and CR0.PE=0. The code/tutorials you link to are not about entering v8066 mode. Entering v8086 mode can only be done on x85-64 from 32-bit protected mode (legacy mode, not long mode) when CPL=0. You don't need VME to exit v8086. A task switch or some combination of instructions that fault the processor while in v8086 mode allows for a protected mode interrupt handler to switch out of v8086 mode and return to executing 32-bit protected mode code. – Michael Petch Feb 23 '19 at 20:24