jmp 100 is a jump to absolute address 100, just like jmp my_label is a jump to the code at my_label. EIP = 100 or EIP = the address of my_label.
(jmp 100 assembles to a jmp rel32 with a R_386_PC32 relocation, asking the linker to fill in the right relative offset from the jmp instruction's own address to the absolute target.)
So in AT&T syntax, you can think of jmp x as sort of like an LEA into EIP.
Or another way to think of it is that code-fetch starts from the specified memory location. Requiring a $ for an immediate wouldn't really make sense, because the machine encoding for direct near jumps uses a relative displacement, not absolute. (http://felixcloutier.com/x86/JMP.html).
Also, indirect jumps use a different syntax (jmp *%eax register indirect or jmp *(%edi, %ecx, 4) memory indirect), so a distinction between immediate vs. memory isn't needed.
But far jump is a different story.
jmp ptr16:32 and jmp m16:32 are both available in 32-bit mode, so you do need to distinguish between ljmp *(%edi) vs. ljmp $100, $100.
Direct far jump (jmp far ptr16:32) does take an absolute segment:offset encoded into the instruction, just like add $123, %eax takes an immediate encoded into the instruction.