UTF-8 characters in HTTP POST query form parameters

Question

I am trying to transfer form data from an Android application to a NodeJs server. My client code is the following (the strings that can contain UTF-8 characters are the values of params):

final HttpPost post = new HttpPost(url);

final MultipartEntityBuilder mpb = MultipartEntityBuilder.create()
.setCharset(Charset.forName("UTF-8"))  // tried with or without this line
.setMode(HttpMultipartMode.BROWSER_COMPATIBLE); // tried with or without this line

for (final Entry<String, String> e : params.entrySet()) {
  mpb.addTextBody(e.getKey(), e.getValue());
}
post.setEntity(mpb.build());
final HttpClient httpClient = new DefaultHttpClient();
final HttpResponse response = httpClient.execute(request);

And my server code is the following:

app.post('/accesspoint', function(req, res) {
  var body = req.body;
  var form = new formidable.IncomingForm();
  form.encoding = 'utf-8';

  form.parse(req, function(err, fields, files) {
    console.log(fields);
    ...

When my input java params has a value containing an UTF-8 character, the log I get server side prints the corresponding value without this character, so it is kind of swallowed at some point. For instance if my input string is "ê", then my server log will print a "" value.

I use a multipart form as I read that it was the best way to send data that can contain non-ASCII characters. Formidable is also apparently the best node package to handle form that can contain UTF-8 characters.

My client side uses Apache HttpClient 4.3.3.

What am I doing wrong?

Have you looked at the headers of the HTTP connection? Is an appropriate header sent to tell the server that the client is sending UTF-8? I'm not familiar with the API, so I don't know if that `setCharset` call causes an appropriate header to be sent. — David Conrad, Aug 19 '14 at 18:15
How do you know the problem is with the client ? Have you tried intercepting the request with a proxy like Charles and checking if the UTF-8 characters are transmitted correctly ? — Deepak Bala, Aug 19 '14 at 18:46

score 0 · Accepted Answer · edited May 23 '17 at 10:33

Ok so I tested with a simple query and the key value ("foo","[ê]") looked at the headers and I saw that my query was still using ISO-8859-1

Content-Disposition: form-data; name="foo"
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
[]

in contradiction to my builder parameter. I found the solution via https://stackoverflow.com/a/21020435/592392 and changed my code to:

for (final Entry<String, String> e : params.entrySet()) {
      mpb.addTextBody(e.getKey(), e.getValue(), ContentType.create("text/plain", Charset.forName("UTF-8")));
    }

And now the server gets the UTF8 chars :)

Anyway, the form builder is quite misleading in my opinion.

UTF-8 characters in HTTP POST query form parameters

1 Answers1