MySQL - utf8 characters no displaying correctly on web frontend

Question

I have a database which has the latin1 default characterset - info obtained by running the following statement:

  SELECT default_character_set_name FROM information_schema.SCHEMATA 

WHERE schema_name = "schemaname";

The default character set for each table and column in this database is set to utf8.

When I look at the data in the tables I can see data is stored as utf8 e.g the currency symbol € is stored in the table as €. Similarly apostraphes are stored as ’ etc.

On the web frontend I have the following meta tag and so the characters render correctly.

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> 

However I'm also seeing a lot of � symbols on the webpage which I don't see inside the database?

When I change the database connection to include the charset utf8 as follows: mysql:host=myhost;dbname=mydatabase;charset=utf8, the diamond symbols disappear but then all the other utf8 characters revert to exactly how they are saved in the database e.g. the € symbol renders as € on the webpage?

Why is this happening?

1. How do I fix this and also change character set to utf8mb4?

Any help appreciated.

* UPDATE *

Tried the following steps:

1. for the database:

   ALTER DATABASE database_name CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;

2. For each table:

   ALTER TABLE table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

3. For each column:

   ALTER TABLE table_name CHANGE column_name column_name VARCHAR(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

Not sure if Step 3 is necessary since when I do SHOW CREATE TABLE after step 2, whilst the definition doesn't display the column charset it does display the default charset for the table as utf8mb4. As a sanity check I did run step 3 on one of the tables columns but it makes no difference - € is being rendered on the page as € with db connection as follows:

`mysql:host=myhost;dbname=mydatabase;charset=utf8mb4`

I had to run the following on each column I wanted converting which seems to fix some issues

 UPDATE tbl_profiles SET profile =
 convert(cast(convert(profile using latin1) as binary) using UTF8MB4);

but still seeing characters such as Iâm and «Âand â¢ rendered on the webpage

Any ideas?

* UPDATE 2 *

After running steps 1 and 2 above I have a table column as follows:

`job_salary` VARCHAR(150) NULL DEFAULT NULL COLLATE 'utf8mb4_unicode_ci',

The following query on this column returns the following result:

SELECT job_salary FROM tbl_jobs WHERE job_id = 2235;

€30,000 plus excellent benefits 

I execute the following statement on this column:

UPDATE tbl_jobs SET job_salary = CONVERT(BINARY(CONVERT(job_salary  USING latin1)) USING utf8mb4);

But I get the following error which means some other record has a invalid utf8mb4

Invalid utf8mb4 character string: '\x8010000 to \x8020000 Per: annum'

Answer 1

First, let's discuss the Mojibake of the Euro sign. All of this applies to both utf8 and utf8mb4, since the Euro is encoded the same way and there is.

It is very likely that the data was initially stored incorrectly. If you can get back to the INSERT program, let's check for:

• The bytes to be stored need to be UTF-8-encoded. What was the client programming language? Where did the data come from?
• The connection when INSERTing and SELECTing text needs to specify utf8 or utf8mb4. Do you have the connection parameters?
• The column needs to be declared CHARACTER SET utf8 (or utf8mb4). It sounds like this was always correct.
• HTML should start with .

What is currently in the table?

SELECT col, HEX(col) FROM ... WHERE ...

A correctly stored Euro sign (€) should have hex E282AC. (Interpreting that as latin1 yields €.

If instead, you see hex C3A2E2809AC2AC, you have "double encoding", and the display is probably €.

I have identified several possible fixes, but have not yet determined which applies in your case. The likely candidate is

• CHARACTER SET utf8mb4 with double-encoding:

To verify it (before fixing it), please do something like:

SELECT col,
       CONVERT(BINARY(CONVERT(col USING latin1)) USING utf8mb4),
       HEX(    
          CONVERT(BINARY(CONVERT(col USING latin1)) USING utf8mb4)
          )
    FROM ...
    WHERE ...

Do not apply a fix on top of another fix. I have struggled for a long time to decipher how character set problems occur and what to do to 'fix' a single problem. But when the wrong fix is applied, I am at a loss to unravel the mess.