So when using bracket notation to access a property, javascript is going to look for that exact key. So by saying obj["name.first"], it's looking for that key literally, so it would match {"name.first": "bob"}, but not {name: {first: "bob"}}.
To get the behavior you want, you have to separate each part manually. So a simple function to get an arbitrary property key might be:
function getKeyAt(obj, key) {
    let result = obj;
    for (const part of key.split(".")) {
        if (!(part in result)) return; // If the key doesn't exist, return.
        result = result[part];
    }
    return result;
}
OR, if you have lodash available, it provides a handy function _.get to do this for you.
Once you have such a function, your code could be modified like so:
handleTableSort = (sortByColumn) => (event) => {
  event.preventDefault();
  const sortByColumn = 'name.first';
  const profileCandidates = this.state.profileCandidates.sort((a, b) => getKeyAt(a, sortByColumn).localeCompare(getKeyAt(b, sortByColumn));
  this.setState({
   profileCandidates: profileCandidates
  })       
}