Q1
I think you are talking about sub-string replacement -- correct me if I am wrong.
There is no reverse thing of %VARIABLE:*search:replac%, but there is a nice work-around, supposing the search part consists of a single character (# in the example below) and it does not appear as the first character in the string:
set "VARIABLE=test#string"
for /F "delims=# eol=#" %%F in ("%VARIABLE%") do echo(%%F
This returns everything in %VARIABLE% before the first occurrence of the # character. (The eol=# portion simply disables the default eol character ;.)
If the delimiter character might also appear at the first position in the string, precede a dummy character temporarily, other than the delimiter (like _, for instance):
set "VARIABLE=test#string"
for /F "delims=#" %%F in ("_%VARIABLE%") do (
    set "VALUE=%%F"
    setlocal EnableDelayedExpansion
    echo(!VALUE:~1!
    endlocal
)
If the search portion consists of multiple characters, things become a bit more complicated, but you could replace it by a single character that does not appear in the part before and use one of the above solutions.
Q2
Magoo already shows the best way to avoid such errors in his answer.
I want to show you why the escaping failed in your approach:
The escaping like ^< and ^> is consumed by setting the variable with set. When expanding it like %VARIABLE%, it appears unescaped, that is why you receive an error. But -- besides quotation -- there are work-arounds:
rem // This sets the variable to `abc<Data>123` literally:
set VARIABLE=abc^<Data^>123
rem // This fails as it expands to `abc<Data>123`:
echo(%VARIABLE%
setlocal EnableDelayedExpansion
rem /* This succeeds dueto delayed expansion, because
rem    special characters are no longer recognised then: */
echo(!VARIABLE!
endlocal
rem // This sets the variable to `abc^<Data^>123` literally:
set VARIABLE=abc^^^<Data^^^>123
rem /* This succeeds as it expands to `abc^<Data^>123`,
rem    hence the poisonous characters are escaped properly: */
echo(%VARIABLE%