No mapping for the Unicode character exists in the target multi-byte code page : how to convert MemoryStream to String

Question

My goal is to be able to convert TMemoryStream to string. I have this code to get the data into the TMemoryStream:

var
  idHttp : TIdHTTPEx;
  url : string;
  slTemp : TStringList;
  memoryStream : TMemoryStream;
begin
  try

    idHttp := TIdHTTPEx.Create(nil);
    slTemp := TStringList.Create;
    memoryStream := TMemoryStream.Create;
    try
      url := GetURL;
      SetParams(slTemp);

  idHttp.Request.Accept := 'application/json, text/javascript, */*; q=0.01';
  idHttp.Request.AcceptEncoding := 'gzip, deflate, br';
  idHttp.Request.AcceptLanguage := 'en-US,en;q=0.9';
  idHttp.Request.CacheControl := 'no-cache';
  idHttp.Request.Connection := 'keep-alive';
  idHttp.Request.ContentLength := 16;
  idHttp.Request.ContentType := 'application/x-www-form-urlencoded; charset=UTF-8';

      idHttp.Post(url, slTemp, memoryStream);
      Result := MemoryStreamToString(memoryStream);
    finally
      memoryStream.Free;
      slTemp.Free;
      idHttp.Free;
    end;

  except on E : Exception do
    begin
      Result := 'e:' + E.Message;
    end;
  end;
end;

And this is my code to convert it to a string:

function MemoryStreamToString(MemoryStream : TMemoryStream): string;
var
  StringStream: TStringStream;
begin
  Result:='';

    StringStream:= TStringStream.Create('', TEncoding.UTF8);
  try
    MemoryStream.Position := 0;
    StringStream.CopyFrom(MemoryStream, MemoryStream.Size);

    Result:= StringStream.DataString;
    Result := Result;
  finally
    FreeAndNil(StringStream);
  end;

end;

My function works fine in most of the conversion but not this. I checked these links: link1, link2 but they are different than my situation. I tried link3 too but still fail.

Any idea how to solve the problem?

Are you sure that the returned data is encoded in UTF-8? You don't seem to check this in your code... — HeartWare, Jun 14 '19 at 07:09
The error message would suggest that the stream does not contain UTF8 encoded text. We don't know what's in your stream. — David Heffernan, Jun 14 '19 at 07:10
one of the request is idHttp.Request.AcceptEncoding := 'gzip, deflate, br'; — Ago, Jun 14 '19 at 07:12
It makes me sad to see so much delphi code like this. A memory stream is just a point to a contiguous block of memory. No need to bring yet another stream object into play, create yet another copy of the data. Decode the payload directly. — David Heffernan, Jun 14 '19 at 07:12
i have updated my code in the question.. i set the charset in my idhttp.request to utf8 but still no avail — Ago, Jun 14 '19 at 07:21
It's time to do some debugging. Inspect the content of the stream. What is it? Stop trying to guess, using trial and error. We can't debug this since we don't have the payload. — David Heffernan, Jun 14 '19 at 07:25
Your changes with AcceptEncoding doesn't mention UTF-8, so we still don't know what encoding the data is being returned in. Check the Response in the HTTP component after the POST to see what the server have told you that the encoding of the response is... — HeartWare, Jun 14 '19 at 08:11
You may try with `Utf8Decode()` function, and don't use streams, but string. — Arnaud Bouchez, Jun 14 '19 at 08:18
@RemyLebeau, big thanks.. its now working. i removed the "ContentType" and get the result as idHttp.Post(url, slTemp). again, thanks.. can you post your comment so that I can accept it as answer? — Ago, Jun 14 '19 at 08:22
@RemyLebeau, I added it because I'm trying to mimick how chrome accessed it.. — Ago, Jun 14 '19 at 08:23

score 4 · Accepted Answer · answered Jun 14 '19 at 14:46

You don't need to decode the raw data manually. Just let TIdHTTP do it for you. The Post() method has an overload that returns a decoded string:

Result := idHttp.Post(url, slTemp);

Also, you need to get rid of this line completely:

idHttp.Request.AcceptEncoding := 'gzip, deflate, br';

Otherwise TIdHTTP will not be able to decode the response correctly if the server decides to send a compressed response. You are manually giving the server permission to do so, but you are not setting up the TIdHTTP.Compressor property so TIdHTTP can handle decompression. Do not set the AcceptEncoding manually unless you are willing and able to manually detect and decode a response that has been actually been encoded in one of the formats you specify. Otherwise, just let TIdHTTP manage the AcceptEncoding property internally based on its actual capabilites.

No mapping for the Unicode character exists in the target multi-byte code page : how to convert MemoryStream to String

1 Answers1