Shouldn't I get some OS error message like
It depends.
Suppose your rebuild command is gcc -o foo t.c.
This command can either open(2) foo for writing, or it can write to a temporary file foo.$uniqsuffix and rename(2) the temporary to foo on success, or it can unlink(2) foo and create and write to a new foo.
Only the first variant -- attempting to write to the original foo would fail with ETXTBSY.
Running strace -fe file gcc -o foo t.c |& grep foo on my (Ubuntu) system shows:
[pid 116892] stat("foo", {st_mode=S_IFREG|0750, st_size=16520, ...}) = 0
[pid 116892] lstat("foo", {st_mode=S_IFREG|0750, st_size=16520, ...}) = 0
[pid 116892] unlink("foo")              = 0
[pid 116892] openat(AT_FDCWD, "foo", O_RDWR|O_CREAT|O_TRUNC, 0666) = 3
[pid 116892] stat("foo", {st_mode=S_IFREG|0640, st_size=16520, ...}) = 0
[pid 116892] chmod("foo", 0750)         = 0
So on this system the linker uses unlink + create new file strategy, and no error is expected.
See this answer for why you can continue debugging the original program even after it has been rebuilt.