Special characters not being rendered properly across different environments

Question

I am working with this library https://github.com/unicode-cldr/cldr-localenames-modern in order to get a list of countries. Some of them have special characters that for some reason the browsers (Chrome and FF) are not rendering properly.

I created this codesandbox https://codesandbox.io/s/zen-jackson-qt9i46?file=/src/App.js where I am using the exact same code that the library is using. Weirdly, in the codesandbox there are no issues at all.

This is the list of countries => https://github.com/unicode-cldr/cldr-localenames-modern/blob/master/main/en/territories.json

On this function I tried 2 different things:

const sortCountryOptions = (options) => {
  return Object.keys(options)
    .filter(key => key.length === 2 && !['EU', 'EZ', 'UN', 'XA', 'XB', 'ZZ'].includes(key))
    .sort((a, b) => {
      const an = (Array.isArray(options[a])) ? options[a][0] : options[a];
      const bn = (Array.isArray(options[b])) ? options[b][0] : options[b];
      return an < bn ? -1 : (an > bn ? 1 : 0);
    })
    .reduce((obj, key) => {
      obj[key] = options[key];
      return obj;
    }, {});
};

const cldr = () => {
  const list = cldrLocaleNames.main.en.localeDisplayNames.territories;
  // when I call this list, I am getting the bad characters. 
  return sortCountryOptions(list);
};

I changed this an < bn ? -1 : (an > bn ? 1 : 0) to this an.localeCompare(bn) but the same keeps going.

On the screenshot below you can check on every column, the name in the middle of each, have problems.

I do not know if this could be something wrong with a webpack config, also, when I deploy to another environment, the error is gone or just different.

And actually the problem is not by the time I render the objects because I detected that this issue on my project happens exactly here: const list = cldrLocaleNames.main.en.localeDisplayNames.territories;

Any ideas?

Please show the frontend code that is a) loading/parsing the script/json files b) the backend code that is *serving* the file. The sorting code you posted has nothing to do with this, it's an encoding mismatch. — Bergi, May 10 '22 at 23:45

score 1 · Answer 1 · answered May 11 '22 at 00:30

Character Encoding

It's a character encoding problem, but you will need to pinpoint the cause. For example, using the wrong meta tag will cause "St. Barthélemy" to appear as "St. BarthÃ©lemy". You can demonstrate this by creating and opening the two simple web pages below, each with a different encoding. Yet, the problem can be something other than a meta tag. For example, how the data is read from the server.

This related SO question might be helpful: utf-8 special characters not displaying

8859 Encoding - displays incorrectly in the browser

<html>
<head>
 <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1" /> 
</head>
<body>
    St. Barthélemy
</body>
</html>

UTF-8 Encoding - displays correctly in the browser

<html>
<head>
 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
    St. Barthélemy
</body>
</html>

Special characters not being rendered properly across different environments

1 Answers1