How to force FastCGI to encode form data as UTF-8 as CGI.pm has option?

Question

Trying to run old CGI-scripts under FastCGI. Printing without extra parameters gives proper output: print $q->div( $q->param("text") )

But when printing out with extra parameters hash for CGI-methods print $q->div( {-id=>"id"}, $q->param("text") ), it ruins UTF-8 formed data ('õäöüžš' -> 'õäöüžš')

It happens only with CGI parameters, in script defined variables work fine (examples 3 and 4). Everything works perfecty under ordinary CGI (with "-utf8"-flag ).

FastCGI-turned example script, called as test.fcgi?text=õäöüžš should give four equal blocks:

#!/usr/bin/perl -w --

use strict;
use CGI::Fast qw(:all);
use locale;
use utf8;

BEGIN {
        binmode(STDIN);                       # Form data
        binmode(STDOUT, ':encoding(UTF-8)');  # HTML
        binmode(STDERR, ':encoding(UTF-8)');  # Error messages
}

my ($q) = ();
my $test = "õäöüžš";

while ($q = new CGI::Fast) {

        print $q->header(-type=>"text/html", -charset=>"utf-8"), 
                $q->start_html(-encoding=>"utf-8");

        print "1: ",
                $q->div(  $q->param('text') ),
                "<br />",
                "2: ",
                $q->div( {-id=>"id"},  $q->param('text') ),
                "<br />",
                "3: ",
                $q->div(  $test ),
                "<br />",
                "4: ",
                $q->div( {-id=>"id"},  $test ),
        $q->end_html();

}

First block is fine, second broken, 3rd and 4th also fine:

Ordinary CGI-example as that gives all 4 right way:

#!/usr/bin/perl -w --

use strict;
use CGI qw(:all -utf8);
use locale;
use utf8;

BEGIN {
        binmode(STDIN);                       # Form data
        binmode(STDOUT, ':encoding(UTF-8)');  # HTML
        binmode(STDERR, ':encoding(UTF-8)');  # Error messages
}

my ($q) = ();
my $test = "õäöüžš";
$q = new CGI;

        print $q->header(-type=>"text/html", -charset=>"utf-8"), 
                $q->start_html(-encoding=>"utf-8");

        print "1: ",
                $q->div(  $q->param('text') ),
                "<br />",
                "2: ",
                $q->div( {-id=>"id"},  $q->param('text') ),
                "<br />",
                "3: ",
                $q->div(  $test ),
                "<br />",
                "4: ",
                $q->div( {-id=>"id"},  $test ),
        $q->end_html();

It seems to me, that with FastCGI form-data has no utf8-flag on and i don't understand, how to properly force it? Under CGI.pm i declare as:

use CGI qw(:all -utf8);

But how with FastCGI?

Answer 1

1) CGI::Fast is a subclass of CGI.pm, so you can specify the same import arguments.

use CGI::Fast (-utf8);

2) FCGI streams are implemented using the older stream API, TIEHANDLE. Applying PerlIO layers using binmode() has no effect. The proper solution would be to encode your data before outputting it, but if thats not an option I can offer this hotpatch:

#!/usr/bin/perl
use strict;
use warnings;
use utf8;

use CGI::Fast (-utf8);
use FCGI      ();
use Encode    ();

my $enc = Encode::find_encoding('UTF-8');
my $org = \&FCGI::Stream::PRINT;
no warnings 'redefine';
local *FCGI::Stream::PRINT = sub {
    my @OUTPUT = @_;
    for (my $i = 1; $i < @_; $i++) {
        $OUTPUT[$i] = $enc->encode($_[$i], Encode::FB_CROAK|Encode::LEAVE_SRC);
    }
    @_ = @OUTPUT;
    goto $org;
};

my $literal = "õäöüžš";

while (my $q = CGI::Fast->new) {
    print $q->header(-type => "text/html", -charset => "UTF-8"),
          $q->start_html(-encoding => "UTF-8"),
          $q->p("Text 1:" . $literal),
          $q->p("Text 2:" . $q->param('text')),
          $q->end_html;
}

Answer 2

With Git 2.27 (Q2 2020), Gitweb should fix the issue.

See commit 2ecfcde (29 Mar 2020) by Julien Moutinho (ju1m).
^{(Merged by Junio C Hamano -- gitster -- in commit 7a8bb6d, 22 Apr 2020)}

gitweb: fix UTF-8 encoding when using CGI::Fast

^{Signed-off-by: Julien Moutinho}

FCGI streams are implemented using the older stream API: TIEHANDLE, therefore applying PerlIO layers using binmode() has no effect to them.

The solution in this patch is to redefine the FCGI::Stream::PRINT function to use UTF-8 as output encoding, except within git_blob_plain() and git_snapshot() which must still output in raw binary mode.

This problem and solution were previously reported back in 2012:

• Git mailing-list: "Gitweb running as FCGI does not print its output in UTF-8"
• this very page

Answer 3

I would do:

use strict;
use CGI;
CGI->compile();
use CGI::Fast;
use utf8;
while (new CGI::Fast) {
    $CGI::PARAM_UTF8=1;# may be this????
    my $q =CGI->new;
    #rest of the code should work
}