Perl XML::Parser encoding problem

Question

I am writing a Perl script that needs to extract some data from an XML file.

The XML file itself is encoded using UTF-8. For some reason, however, what I extract from the file ends up being encoded as ISO-8859-1. The documentation states that whatever is passed to my handlers should be UTF-8, but it just isn't.

The parser is basically something like this:

my $parser = XML::Parser->new( Handlers => {
    # Some unrelated handlers here
    Char => sub {
        my ( $expat, $string ) = @_;
        if ( exists $data->{$curId}{$curField} ) {
            $data->{$curId}{$curField} .= $string;
        } else {
            $data->{$curId}{$curField} = $string;
        }
    } ,
} );

I have tried the following variants for actually parsing:

file parsed directly through $parser->parsefile, no options;
file parsed directly through $parser->parsefile, with the ProtocolEncoding option;
file opened using open( $handle , "<file.xml" ) then parsed through $parser->parse;
file opened using open( $handle , '<:utf8' , "file.xml" ) then parsed through $parser->parse.

In addition, I have tried each version with and without the <?xml encoding="utf-8"?> header in the file.

In all cases, what ends up in $data->{$curId}{$curField} is encoded using ISO-8859-1.

What am I doing wrong?

Just tried, seems to work fine for me (ActivePerl 5.12 on WinXP). You can check if `$string` is marked as UTF-8 with `use utf8; utf8::is_utf8($string)`. If you get true, problem is likely in processing that follows. — bvr, Mar 19 '11 at 11:33
Are you sure it's in ISO-8859-1? How did you test this? Because if you used `print` on any file not open in utf-8 mode (eg STDOUT or STDERR), and your characters are in ISO-8859-1 range, then by default Perl will convert them to ISO-8859-1 (for compatibility sake). You can use `binmode STDOUT, ":utf8"` to make sure the conversion doesn't happen on STDOUT. — mirod, Mar 19 '11 at 11:36
That was indeed the problem - I didn't know I had to change STDOUT's mode and assumed it was following my locale settings. Thanks guys. — E.Benoît, Mar 19 '11 at 11:54

score 1 · Accepted Answer · edited May 23 '17 at 10:28

I know you already found an answer from Michel in the comments, but I'll add a few things. With any encoding, you have to be strict about knowing what you're taking in and what you are sending out. If you need something, don't rely on the environment; eventually someone else will use your program and have a screwed-up environment.

When you are reading a file, don't use the ':utf8' layer. That doesn't care if the octets are actually UTF-8:

 open my $fh, '<:encoding(UTF-8)', $filename or ...;

No matter what you think your output handle is, set it explicitly. There are a variety of ways to do this:

 use open ':encoding(utf8)';

From the command-line, you can use the -C switch with the S flag to make the standard handles UTF-8:

 perl -CS input.xml

Tom Christiansen has a long list of things you need to pay attention to.

score 0 · Answer 2 · answered Mar 19 '11 at 11:25

0

Does $data->{$curId}{$curField} have utf8 flag on?

If you concatenate a string with the utf8 flag on with a string that has utf8 flag off, Perl converts the latter to Unicode. This is the usual source of problems.

answered Mar 19 '11 at 11:25

Eugene Yarmash

142,882
41
325
378

Perl XML::Parser encoding problem

2 Answers2