EDIT: I started writing the answer before JavaScript was specified. Will leave it as for the record as it fully explains the regex.
In short: In JavaScript I cannot think of a reliable solution. In other engines there are several options: 
- Recursion (on which I will expand below)
- Balancing group (.NET)
For solutions 2 (which anyhow won't work in JS either), I'll refer you to the example in this question
Recursive Regex
In Perl, PCRE (e.g. Notepad++, PHP, R) and the Matthew Barnett's regex module for Python, you can use:
^({(?:[^{}]++|(?1))*})$
The idea is to match exactly one set of nested braces. Anything more makes the regex fail.
See what matches and fails in the Regex Demo.
Explanation
- The ^anchor asserts that we are at the beginning of the string
- The outer parentheses define Group 1 (or Subroutine 1)
- {match the opening brace
- (?: ...  )*zero or more times, we will...
- [^{}]++match any chars that are not- {or- }
- OR |
- (?1)repeat the expression of subroutine 1
- }match closing brace
- The $anchor asserts that we are at the end of the string. Therefore,