UTF-8 converting data

Question

I use ZF2 and a mysql database. The problem is the data are latin1 encoded, but for the moment I don't have to change that. So I use a lot of tricks to force my PDO driver to have my results on UTF-8 :

This one in local.php :

return array(
    'db' => array(
        'username' =>  $dbParams['username'],
        'dsn'      => 'mysql:dbname='.$dbParams['database'].';host='.$dbParams['hostname'].';charset=utf-8',
        'password' => '',
        'charset'  => 'utf-8',
    ),
);

And this one in global.php

return array(
    'db' => array(
        'charset'  => 'utf-8',
        'driver' => 'Pdo',
        'driver_options' => array(
            PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES UTF8",
            PDO::MYSQL_ATTR_INIT_COMMAND => "SET CHARACTER SET UTF8",
         ),
    ),
    'service_manager' => array(
        'factories' => array(
            'Zend\Db\Adapter\Adapter' => 'Zend\Db\Adapter\AdapterServiceFactory',
        )
    ),
    'translator' => array(
        'locale' => 'fr_FR'
    ),
);

But when I do :

echo "ENCODING=".mb_detect_encoding($myDataBaseRetrievedVar);

This tells me : ASCII ! And for special characters I can't do a json_encode.

Is the database itself utf8 encoded? Try running the query `SHOW CREATE TABLE yourTableHere`. — Qirel, Oct 13 '15 at 08:54
No the database is Latin1 encoded, as I told so. But I mustn't touch it for the moment ... — Amelie, Oct 13 '15 at 08:56
The database *charset*, database *collation* (charset is not the same as collation) and SQL connection should all be the same charset. — Qirel, Oct 13 '15 at 08:57
The MySQL encoding is called `utf8`, not `utf-8`. How about you start there? — deceze, Oct 13 '15 at 09:08
Please provide some more actual data to aid debugging here. Show us a value, what encoding it's detected as, what its `bin2hex()` value is, and what code exactly you use to `json_encode` it. — deceze, Oct 13 '15 at 09:37
Okay. I have a problem with "blob" data ... I think this the problem. I have a field "blob" if a deactivate it, my json_encode works well. — Amelie, Oct 13 '15 at 09:53
Okay. In fact, with this blob field, I have both UTF-8 and ASCII data it is strange ... — Amelie, Oct 13 '15 at 10:27
`SELECT col, HEX(col) FROM tbl WHERE ...` -- With that we can possibly diagnose what to do next. — Rick James, Oct 26 '15 at 01:32

Your Common Sense · Answer 1 · 2015-10-13T09:50:42.890

2

First, for mysql, encoding is called utf8, not utf-8.

Second, basic latin1 characters are the same in utf-8. Means 'Amelie' in utf-8 is the same string as 'Amelie' in latin1 as well as ASCII. Means having 'Amelie' in ASCII you have it in utf-8 as well.

Third, latin1 supports limited amount of "special characters" and you simply cannot expect from a single-byte encoding the same number of characters supported as from the multi-byte one.

A meta note: if you have particular problem with particular function, you have to ask the question regarding this particular problem, not something else. Otherwise no answer will be good for you.

In case you have a problem with json_encode(), you have to supply with your question:

the code
the exact error message
the data sample

And only after solving this problem, you may proceed to the database interaction, that may be the cause for the problem. In programming, no solution can be found based on a guess.

edited Oct 13 '15 at 09:50

answered Oct 13 '15 at 09:11

Your Common Sense

156,878
40
214
345

Okay, so tell me why my json_encode does not work ... and json_last_error_msg() tells me that my error is UTF-8 malformed strings ... – Amelie Oct 13 '15 at 09:19
Furthermore, my question doesn't deserve a -1 ... – Amelie Oct 13 '15 at 09:24
@Amelie You detect your encoding as ASCII, yet `json_encode` complains? That's... weird, to say the least. – deceze Oct 13 '15 at 09:27

UTF-8 converting data

1 Answers1