Would anyone know where the best "official" Intel documentation regarding the low level architecture/assembler internals would be? How the ESP works, the various registers- fetching values from RAM etc?
The Intel website is massive and I could find Agner Fog's website- but I was expecting a mammoth document for this.
The absolute definitive reference is the Intel 64 and IA-32 Architectures Software Developer Manuals, they're a great resource to understand the x86 architecture from a programmer's point of view (assuming you're comfortable with assembly already).
I would recommend that you download the combined volumes 1 through 3C (first download link on that page). Volume 1 is a great overview of Intel's x86(-64) architecture, including various details regarding pipeline depths, register sizes, and available components in their processors. If you need to refer to how specific instructions work, see Vol. 2, Ch. 3-4 (I believe binary instruction encoding is covered later in that volume as well).
For how the assembler itself works, that all depends on what assembler you're using in the first place. You can easily find the assembler details for NASM/GAS on their respective websites, and there's tons of resources for MASM(32). However, this is all independent of how the CPU works, so I believe what you're asking for should be covered entirely by the Intel documentation linked to above.
A good overview about the available resources is the StackOverflow x86 tag wiki.
If you like to directly refer to Intel, the Intel® 64 and IA-32 Architectures Software Developer Manuals will satisfy all your needs. It's about 3800+ pages...
To analyze the performance of your software, the Intel® Architecture Code Analyzer is very useful.
And if your processor is outdated (like one that does not support AVX-512 :-), you can use the Intel® Software Development Emulator.
