ASP.NET Core 2.1 API POST body is null when called using HttpWebRequest, seems it can't be parsed as JSON

Question

I'm facing a strange bug, where .NET Core 2.1 API seems to ignore a JSON body on certain cases.

I advised many other questions (e.g this one, which itself references others), but couldn't resolve my problem.

I have something like the following API method:

[Route("api/v1/accounting")]
public class AccountingController
{                                            sometimes it's null
                                                       ||
    [HttpPost("invoice/{invoiceId}/send")]             ||
    public async Task<int?> SendInvoice(               \/
         [FromRoute] int invoiceId, [FromBody] JObject body
    ) 
    {
        // ...                                                                   
    }
}

And the relevant configuration is:

public IServiceProvider ConfigureServices(IServiceCollection services)
{
     services
       .AddMvcCore()
       .AddJsonOptions(options =>
        {
           options.SerializerSettings.Converters.Add(new TestJsonConverter());
        })
       .AddJsonFormatters()
       .AddApiExplorer();
     
     // ...
}

Where TestJsonConverter is a simple converter I created for testing why things doesn't work as they should, and it's simple as that:

public class TestJsonConverter : JsonConverter
{
    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        var token = JToken.Load(reader);
        return token;
    }
    public override bool CanRead
    {
        get { return true; }
    }
    public override bool CanConvert(Type objectType)
    {
        return true;
    }
    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        throw new NotImplementedException("Unnecessary (would be neccesary if used for serialization)");
    }
}

Calling the api method using Postman works, meaning it goes through the JSON converter's CanConvert, CanRead, ReadJson, and then routed to SendInvoice with body containing the parsed json.

However, calling the api method using HttpWebRequest (From a .NET Framework 4, if that matters) only goes through CanConvert, then routes to SendInvoice with body being null.

The request body is just a simple json, something like:

{
    "customerId": 1234,
    "externalId": 5678
}

When I read the body directly, I get the expected value on both cases:

using (var reader = new StreamReader(context.Request.Body))
{
   var requestBody = await reader.ReadToEndAsync(); // works
   var parsed = JObject.Parse(requestBody);
}

I don't see any meaningful difference between the two kinds of requests - to the left is Postman's request, to the right is the HttpWebRequest:

To be sure, the Content-Type header is set to application/json. Also, FWIW, the HttpWebRequest body is set as follows:

using(var requestStream = httpWebRequest.GetRequestStream())
{
    JsonSerializer.Serialize(payload, requestStream);
}

And called with:

var response = (HttpWebResponse)request.GetResponse();

Question

Why does body is null when used with HttpWebRequest? Why does the JSON converter read methods are skipped in such cases?

"Calling the api method using Postman works" How postman can use your custom converter? It must be inside of the api and so it doesn't matter what do you use it uses the same converter. — Serge, Dec 09 '21 at 17:17
@serge, Postman doesn't "use" the converter. The app is configured to use the converter on every request, which it does. The difference between the two request sources is that when coming from Postman, all of the converter's methods are executed, as expected. This is in contrary to using HttpWebRequest. — OfirD, Dec 09 '21 at 17:28
Controller action doesn' t know who sent the request - a postman, or somebody else — Serge, Dec 09 '21 at 17:33
Yes, this is my point, the converter has nothing to do with this. Your webrequest is the problem. It is time to use http client. — Serge, Dec 09 '21 at 17:46
@serge the converter has a different behavior for each case. Of course it has nothing to do with the *problem*, rather it's a *symptom* which could be helpful for indicating the actual problem. — OfirD, Dec 09 '21 at 18:05

score 0 · Accepted Answer · answered Dec 12 '21 at 10:35

The problem was in the underlying code of the serialization. So this line:

JsonSerializer.Serialize(payload, requestStream);

Was implemented using the default UTF8 property:

public void Serialize<T>(T instance, Stream stream)
{
   using(var streamWriter = new StreamWriter(stream, Encoding.UTF8) // <-- Adds a BOM
   using(var jsonWriter = new JsonTextWriter(streamWriter))
   {
       jsonSerializer.Serialize(jsonWriter, instance); // Newtonsoft.Json's JsonSerializer
   } 
}

The default UTF8 property adds a BOM character, as noted in the documentation:

It returns a UTF8Encoding object that provides a Unicode byte order mark (BOM). To instantiate a UTF8 encoding that doesn't provide a BOM, call any overload of the UTF8Encoding constructor.

It turns out that passing the BOM in a json is not allowed per the spec:

Implementations MUST NOT add a byte order mark (U+FEFF) to the beginning of a networked-transmitted JSON text.

Hence .NET Core [FromBody] internal deserialization failed.

Lastly, as for why the following did work (see demo here):

using (var reader = new StreamReader(context.Request.Body))
{
   var requestBody = await reader.ReadToEndAsync(); // works
   var parsed = JObject.Parse(requestBody);
}

I'm not very sure. Certainly, StreamReader also uses UTF8 property by default (see remarks here), so it shouldn't remove the BOM, and indeed it doesn't. Per a test I did (see it here), it seems that ReadToEnd is responsible for removing the BOM.

For elaboration:

ASP.NET Core 2.1 API POST body is null when called using HttpWebRequest, seems it can't be parsed as JSON

1 Answers1