To match a substring inside parentheses excluding any inner parentheses you may use
\(([^()]*)\)
pattern. See the regex demo.
In JavaScript, use it like
var rx = /\(([^()]*)\)/g;
Pattern details
\( - a ( char 
([^()]*) - Capturing group 1: a negated character class matching any 0 or more chars other than ( and ) 
\) - a ) char. 
To get the whole match, grab Group 0 value, if you need the text inside parentheses, grab Group 1 value.
Most up-to-date JavaScript code demo (using matchAll):
const strs = ["I expect five hundred dollars ($500).", "I expect.. :( five hundred dollars ($500)."];
const rx = /\(([^()]*)\)/g;
strs.forEach(x => {
  const matches = [...x.matchAll(rx)];
  console.log( Array.from(matches, m => m[0]) ); // All full match values
  console.log( Array.from(matches, m => m[1]) ); // All Group 1 values
});
 
 
Legacy JavaScript code demo (ES5 compliant):
var strs = ["I expect five hundred dollars ($500).", "I expect.. :( five hundred dollars ($500)."];
var rx = /\(([^()]*)\)/g;
for (var i=0;i<strs.length;i++) {
  console.log(strs[i]);
  // Grab Group 1 values:
  var res=[], m;
  while(m=rx.exec(strs[i])) {
    res.push(m[1]);
  }
  console.log("Group 1: ", res);
  // Grab whole values
  console.log("Whole matches: ", strs[i].match(rx));
}