To do a normal relative jump that results in RIP = label, use jmp label.
The assembler calculates the relative displacement.
(And yes, you must use jmp not jmpq. An operand-size suffix for jmp implies an indirect jump in AT&T syntax, unlike with calll / callq, which is an insane and confusing bad design.)
Relative direct jumps work the same as they have since 8086, and don't have anything to do with the RIP-relative data addressing mode that was new in x86-64. (How do RIP-relative variable references like "[RIP + _a]" in x86-64 GAS Intel-syntax work? also covers how the AT&T syntax works for 4(%rip) vs. label(%rip) for data addressing.)
You don't want to load a new RIP from a function pointer, so don't use a memory addressing mode. A relative jmp doesn't access memory itself; the memory at label only gets accessed by code-fetch after the jmp finishes. (Real CPUs are pipelined with branch prediction and stuff, but still maintain the illusion of instructions fully executing before the next one starts, e.g. a relative jmp to an unmapped page can fully execute without itself faulting, only the next code-fetch operation faults with saved RIP = the new location.)
Then jmpq 0x4(%rip) jumps to 4+rip
No, it doesn't. If you actually assemble that, it warns indirect jmp without '*'. Disassembly shows
ff 25 04 00 00 00 jmp *0x4(%rip)
i.e. it loads a pointer from memory into RIP, from the qword that starts 4 bytes after the end of the jmp instruction. It doesn't set RIP = RIP+4, that would be jmp .+2 + 4 (because a short jump is 2 bytes long, or if you put a label after the jmp, jmp end_of_insn + 4)