Doesn't 64-bit kernel support IA-32 application with .code64 assembler?

Question

x86 application(build with gcc -m32) supports Assembly 64-bit code with .code64, this means that an x86 application could use a 64-bit register. But from kernel side, the application just a IA-32 application.

For example, I can link below symbol 64bit_test to an x86 application.

ENTRY(64bit_test)
    .code64;
    push %r8
    push %r12
END(64bit_test)

When the kernel setup signal handler, the kernel only saved 32-bit registers without 64-bit registers, does 64-bit register context missing? I think this is incorrect because 64-bit register is used and should be saved and restored later.

if (is_ia32_frame(ksig)) {
        if (ksig->ka.sa.sa_flags & SA_SIGINFO)
            return ia32_setup_rt_frame(usig, ksig, cset, regs);
        else
            return ia32_setup_frame(usig, ksig, cset, regs);
    } else if (is_x32_frame(ksig)) {
        return x32_setup_rt_frame(ksig, cset, regs);
    } else {
        return __setup_rt_frame(ksig->sig, ksig, set, regs);
    }


static int ia32_setup_sigcontext(struct sigcontext_32 __user *sc,
                 void __user *fpstate,
                 struct pt_regs *regs, unsigned int mask)
{
    int err = 0;

    put_user_try {
        put_user_ex(get_user_seg(gs), (unsigned int __user *)&sc->gs);
        put_user_ex(get_user_seg(fs), (unsigned int __user *)&sc->fs);
        put_user_ex(get_user_seg(ds), (unsigned int __user *)&sc->ds);
        put_user_ex(get_user_seg(es), (unsigned int __user *)&sc->es);

        put_user_ex(regs->di, &sc->di);
        put_user_ex(regs->si, &sc->si);
        put_user_ex(regs->bp, &sc->bp);
        put_user_ex(regs->sp, &sc->sp);
        put_user_ex(regs->bx, &sc->bx);
        put_user_ex(regs->dx, &sc->dx);
        put_user_ex(regs->cx, &sc->cx);
        put_user_ex(regs->ax, &sc->ax);
        put_user_ex(current->thread.trap_nr, &sc->trapno);
        put_user_ex(current->thread.error_code, &sc->err);
        put_user_ex(regs->ip, &sc->ip);
        put_user_ex(regs->cs, (unsigned int __user *)&sc->cs);
        put_user_ex(regs->flags, &sc->flags);
        put_user_ex(regs->sp, &sc->sp_at_signal);
        put_user_ex(regs->ss, (unsigned int __user *)&sc->ss);

        put_user_ex(ptr_to_compat(fpstate), &sc->fpstate);

        /* non-iBCS2 extensions.. */
        put_user_ex(mask, &sc->oldmask);
        put_user_ex(current->thread.cr2, &sc->cr2);
    } put_user_catch(err);

    return err;
}

I expect 64-bit registers r8 to r15 should be saved in sigcontext and later restored, but from the code, r8 to r15 is missing.

Answer 1

TL:DR: no, that's not what .code64 does, and no Linux doesn't support 32-bit processes that far-jump to 64-bit user-space.

---

.code64 just lets you put 64-bit machine code inside a 32-bit object file / executable. e.g. if you wanted to write a 32-bit program that patched a 64-bit executable, and wanted to have the assembler generate that data for you even though it would never be executed inside the 32-bit program.

Or if you were writing your own kernel that started in 16 or 32-bit mode, and switched to 64-bit mode, you'd use .code64 for the part that your kernel jumps with CS referring to a 64-bit code segment.

---

Decoding machine code as 64-bit instead of 32-bit requires putting the CPU in a different mode. x86 machine code does not support mixing 32 and 64-bit machine code without a mode-switch. There isn't enough coding space left for that. The encodings are very similar but with some opcodes having a different default operand-size in 64-bit mode (e.g. stack ops) e.g. push %eax and push %rax have the same 1-byte opcode.

Your .code64; ; push %r8 test actually creates 32-bit machine code for inc %eax (the REX prefix) and push %eax. Yes it assembles and runs, but as different instructions. Single-step it with GDB in layout reg to see disassembly according to the actual mode the CPU is in, not source.

The differences include 64-bit long mode repurposing 1-byte inc/dec (0x40..4f) opcodes as REX prefixes. e.g. x86-32 / x86-64 polyglot machine-code fragment that detects 64bit mode at run-time?

---

Note that this is very different from 16 vs. 32. 16-bit code can use an operand-size prefix in 16-bit mode to access 32-bit registers and addressing modes. e.g. mov eax, 1234 assembles just fine in .code16 (with an operand-size prefix), or in .code32 (without a prefix).

But you can't do add rax, rdx outside of .code64 because there's no way to run it without switching the CPU to a different mode. (Modes are selected by the GDT / LDT entry that CS is pointing to).

---

You can in theory jmpl (far jmp) in user-space to a different code segment in your user-space process to switch from "compat mode" (32-bit mode under a 64-bit kernel) to full 64-bit mode. You'd have to know what CS value to use, but most OSes have some "well-known" constant values for their 32-bit and 64-bit user-space (CPL=3) code segments.

If that sounds incredibly arcane and complex, yeah that's my point.

There's basically zero support (from OS system calls and context switches, dynamic linker, and toolchains) for switching modes inside a process. It's generally a terrible idea, don't do it.

e.g. as you noticed, the kernel only saves / restores the legacy IA32 state for a process that started as 32-bit when delivering a signal, so if it had far-jumped to 64-bit user-space the signal handler would corrupt high registers. (r8..r11 are call-clobbered in the x86-64 System V ABI).

Semi-related: What happens if you use the 32-bit int 0x80 Linux ABI in 64-bit code?