The problem is that you match $( and ) that may mismatch. You only need to match ) after $( and the $( that is followed with ).
Solution 1: Using Matcher#appendReplacementCallback
You can use a simple regex to match $(...) strings and capture the text inside them to get the token from the map, and perform the replacements "on the go" while matching:
SortedMap<String, String> map = new TreeMap<String, String>();
map.put("test", "REPLACE");
String update = "$(test) (test) (test2)";
Matcher m = Pattern.compile("\\$\\(([^)]*)\\)").matcher(update);
StringBuffer sb = new StringBuffer();
while (m.find()) {
m.appendReplacement(sb, map.getOrDefault(m.group(1), m.group(1)));
}
m.appendTail(sb);
System.out.println(sb);
See the IDEONE demo
Solution 2: Using Lambda with Split and Lookarounds
Disclaimer: This approach is only good for not-so-long strings.
You can also use lookarounds. With the lookahead, there is no problem, it is infinite width. With lookbehind, we can rely on the constrained width lookbehind (using a limiting quantifier instead of + or *):
SortedMap<String, String> map = new TreeMap<String, String>();
map.put("test", "REPLACE");
String update = Arrays.stream("$(test) (test) (test2)"
.split("\\$\\((?=[^)]*\\))|(?<=\\$\\([^(]{0,1000})\\)"))
.map(token -> map.getOrDefault(token, token))
.collect(Collectors.joining(""));
System.out.println(update); // => REPLACE (test) (test2)
See the IDEONE demo
The regex now reads:
\$\((?=[^)]*\)) - match a $( that is followed with 0+ characters other than ) and then a )
| - or
(?<=\$\([^(]{0,1000})\) - match a ) that is preceded with 0-1000 (that should be enough) characters other than ( that are preceded with $(.