(%sp) would be an AT&T syntax addressing mode.  (Invalid because 16-bit addressing modes can't use SP directly, only BP|BX + SI|DI NASM x86 16-bit addressing modes;  that's also the reason mov [sp], bx is invalid.)
In NASM syntax, square brackets [] mean a memory operand.
In NASM, the parens () around SP are removed just like any compile-time expression,
so mov (sp), bx assembles to  89DC  mov sp,bx.  Try it yourself by assembling and using ndisasm on the output.  (Or assemble into -felf32 and use objdump)
This is a mov between two registers, overwriting the stack pointer.  Very likely not what you want, and totally different from storing to memory with mov [bp], bx or whatever.
In NASM, you might use parens when writing something like mov ax, (1+3) * 4 so NASM's expression parser handles parens, and apparently having a register name inside parens doesn't change anything.
I only mentioned AT&T syntax at the top of this answer because that and Plan9/Go syntax are the only time you'd normally put a register name inside parens; it's just confusing in NASM syntax; don't do it.