Regular expressions are usually used to define a pattern that a subject should match. What you are asking for is to define a pattern that a subject should not match. Inverse matching is not a built in feature in regex, however it is possible to mimic this behaviour by using a negative lookaround.
The following expression uses a negative lookahead (?!...) to match what you need:
^((?!(\\\*)|(\\\|)|(\\\^)|(\\\~)).)*$
This SO answer provides an excellent explanation on how the negative lookahead achieves the inverse regex match behaviour.
I guess this also answers your questions about how to escape the backslash character as well. You just double escape as I have done above.
About Client + Server side validation in Asp.Net WebForms
The RegularExpressionValidator control will do both server and client side validation for you so long as you have EnableClientScript set to true. This will cause error messages to display when the ControlToValidate loses focus (client side) or when the page is posted back (server side).