Since you are using POSIX ERE syntax (-r option) a capturing group is defined with a pair of unescaped ( and ). Besides, since you have a ' inside the pattern, you need to use doulbe quotes around the s/.../.../ command.
Use
sed -i.bak -r "s/^(\S*HTTPS*_SERVER\S*)(\s*'https*:\/\/)(\S*)('\);)$/\1\2test.com\4/" file.txt
See an online demo.
Note that you may use delimiters other than / to avoid escaping / symbols, in the pattern (say, you may use "s,<pattern>,<replacement>,", as there are no commas in the pattern itself). Besdies, as per this thread, if you really need to use a single quoted regex definition, you may replace a literal ' in the pattern with a \x27 entity.
Also, if you need not insert any text in between Group 1 and Group 2, you may merge the two group patterns into one, and - as you are removing Group 3, you do not even need to capture it.
So, an alternative could look like
sed -r 's,^(\S*HTTPS*_SERVER\S*\s*\x27https*://)\S*(\x27\);)$,\1test.com\2,' file.txt
See another online demo.