How do I write a float like "82.0" with the ".0" intact, using Utf8JsonWriter?

Question

I've been struggling to write a double such as 82.0 using Utf8JsonWriter.

By default, the method WriteNumberValue method takes a double and formats it for me, and the format (which is the standard 'G' format) omits the ".0" suffix. I can't find a way to control that.

By design it seems that I can't just write a raw string to Utf8JsonWriter, but I found a workaround: to create a JsonElement and call JsonElement.WriteTo`. This calls a private method in Utf8JsonWriter and writes the string directly into it.

With this discovery, I made what feels like a very hacky and inefficient implementation.

open System.Text.Json

void writeFloat(Utf8JsonWriter w, double d) {
  String floatStr = f.ToString("0.0################")
  JsonElement jse = JsonDocument.Parse(floatStr).RootElement
  jse.WriteTo(w)
}

I need to format a double anyway so that's fine, but parsing it, creating a jsonDocument and a JsonElement, just to be able to find a way to call a protected method, seems really wasteful. However, it does work (I wrote it in F# and translated to C#, apologies if I made an error in the syntax).

Is there a better way? Some potential solutions that come to mind (I'm new to dotnet so I'm not sure what's possible here):

• is there a way to access the private API directly? I thought subclassing Utf8Writer might work but it's a sealed class.
• can I instantiate a JsonElement directly without the whole parsing rigmarole?

As for why this is necessary: I need to force integer values to be written with a .0 appended because there is an extremely specific format that I need to interact with, which distinguishes between integer and floating-point JSON values. (I'm OK with exponential format as that's clearly a float).

Answer 1

Your requirements are to create a JsonConverter<double> satisfying the following:

• When formatting double values in fixed format, when the value is an integer a .0 fractional portion must be appended.

• No change when formatting in exponential format.

• No change when formatting non-finite doubles such as double.PositiveInfinity.

• No requirement to support JsonNumberHandling options WriteAsString or AllowReadingFromString.

• No intermediate parsing to JsonDocument.

In .NET 6 and later you may format your double manually and write it out with Utf8JsonWriter.WriteRawValue(). The following converter functions as required:

public class DoubleConverter : JsonConverter<double>
{
    const bool skipInputValidation = true; // Set to true to prevent intermediate parsing.  Be careful to ensure your raw JSON is well-formed.

    public override void Write(Utf8JsonWriter writer, double value, JsonSerializerOptions options)
    {
        Span<byte> utf8bytes = stackalloc byte[33]; // JsonConstants.MaximumFormatDecimalLength + 2, https://github.com/dotnet/runtime/blob/v6.0.11/src/libraries/System.Text.Json/src/System/Text/Json/JsonConstants.cs#L85
        if (!double.IsFinite(value))
            // Utf8JsonWriter does not take into account JsonSerializerOptions.NumberHandling so we have to make a recursive call to serialize
            JsonSerializer.Serialize(writer, value, new JsonSerializerOptions { NumberHandling = options.NumberHandling });
        else if (Utf8Formatter.TryFormat(value, utf8bytes.Slice(0, utf8bytes.Length-2), out var bytesWritten))
        {
            // Check to make sure the value was actually serialized as an integer and not, say, using scientific notation for large values.
            if (IsInteger(utf8bytes, bytesWritten))
            {
                utf8bytes[bytesWritten++] = (byte)'.';
                utf8bytes[bytesWritten++] = (byte)'0';
            }   
            writer.WriteRawValue(utf8bytes.Slice(0, bytesWritten), skipInputValidation);
        }
        else // Buffer was too small?
            writer.WriteNumberValue(value);
    }
    
    static bool IsInteger(Span<byte> utf8bytes, int bytesWritten)
    {
        if (bytesWritten <= 0)
            return false;
        var start = utf8bytes[0] == '-' ? 1 : 0;
        for (var i = start; i < bytesWritten; i++)
            if (!(utf8bytes[i] >= '0' && utf8bytes[i] <= '9'))
                return false;
        return start < bytesWritten;
    }
    
    public override double Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) => 
        // TODO: Handle "NaN", "Infinity", "-Infinity"
        reader.GetDouble();
}

Notes:

• For performance I format the value to a utf8 stackalloc'ed byte array using Utf8Formatter, then add the .0 if required, then finally write using skipInputValidation = true. Doing so should result in best performance, as Utf8JsonWriter is designed to write directly to utf8 buffers or streams rather than to utf16 text writers whose contents are subsequently encoded to utf8.

• Utf8Formatter produces locale-invariant output, but if you use a ToString() method such as f.ToString("0.0################"), be sure to do so in the invariant locale like so:

  f.ToString("0.0################", NumberFormatInfo.InvariantInfo);
  

  This guarantees that the correct JSON decimal separator . will be used even in locales where a comma is used.

• The double.IsFinite(value) check is intended to serialize non-finite values like double.PositiveInfinity correctly. Upon experimentation I have found that Utf8JsonWriter.WriteNumberValue(value) throws unconditionally for these types of value so the serializer must be called to properly handle them when JsonNumberHandling.AllowNamedFloatingPointLiterals is enabled.

Demo fiddle #1 here.

In .NET 5 and earlier Utf8JsonWriter.WriteRawValue() do not exist, so, as suggested by mjwills in comments, you can convert the double to a decimal with the required fractional component, then write that to JSON as follows:

public class DoubleConverter : JsonConverter<double>
{
    // 2^49 is the largest power of 2 with fewer than 15 decimal digits.  
    // From experimentation casting to decimal does not lose precision for these values.
    const double MaxPreciselyRepresentedIntValue = (1L<<49);

    public override void Write(Utf8JsonWriter writer, double value, JsonSerializerOptions options)
    {
        bool written = false;
        // For performance check to see that the incoming double is an integer
        if ((value % 1) == 0)
        {
            if (value < MaxPreciselyRepresentedIntValue && value > -MaxPreciselyRepresentedIntValue)
            {
                writer.WriteNumberValue(0.0m + (decimal)value);
                written = true;
            }
            else
            {
                // Directly casting these larger values from double to decimal seems to result in precision loss, as noted in  https://stackoverflow.com/q/7453900/3744182
                // And also: https://learn.microsoft.com/en-us/dotnet/api/system.convert.todecimal?redirectedfrom=MSDN&view=net-5.0#System_Convert_ToDecimal_System_Double_
                // > The Decimal value returned by Convert.ToDecimal(Double) contains a maximum of 15 significant digits.
                // So if we want the full G17 precision we have to format and parse ourselves.
                //
                // Utf8Formatter and Utf8Parser should give the best performance for this, but, according to MSFT, 
                // on frameworks earlier than .NET Core 3.0 Utf8Formatter does not produce roundtrippable strings.  For details see
                // https://github.com/dotnet/runtime/blob/eb03e0f7bc396736c7ac59cf8f135d7c632860dd/src/libraries/System.Text.Json/src/System/Text/Json/Writer/Utf8JsonWriter.WriteValues.Double.cs#L103
                // You may want format to string and parse in earlier frameworks -- or just use JsonDocument on these earlier versions.
                Span<byte> utf8bytes = stackalloc byte[32];
                if (Utf8Formatter.TryFormat(value, utf8bytes.Slice(0, utf8bytes.Length-2), out var bytesWritten)
                    && IsInteger(utf8bytes, bytesWritten))
                {
                    utf8bytes[bytesWritten++] = (byte)'.';
                    utf8bytes[bytesWritten++] = (byte)'0';
                    if (Utf8Parser.TryParse(utf8bytes.Slice(0, bytesWritten), out decimal d, out var _))
                    {
                        writer.WriteNumberValue(d);
                        written = true;
                    }   
                }
            }
        }
        if (!written)
        {
            if (double.IsFinite(value))
                writer.WriteNumberValue(value);
            else
                // Utf8JsonWriter does not take into account JsonSerializerOptions.NumberHandling so we have to make a recursive call to serialize
                JsonSerializer.Serialize(writer, value, new JsonSerializerOptions { NumberHandling = options.NumberHandling });
        }
    }
    
    static bool IsInteger(Span<byte> utf8bytes, int bytesWritten)
    {
        if (bytesWritten <= 0)
            return false;
        var start = utf8bytes[0] == '-' ? 1 : 0;
        for (var i = start; i < bytesWritten; i++)
            if (!(utf8bytes[i] >= '0' && utf8bytes[i] <= '9'))
                return false;
        return start < bytesWritten;
    }
    
    public override double Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) => 
        // TODO: Handle "NaN", "Infinity", "-Infinity"
        reader.GetDouble();
}

Notes:

• This works because decimal (unlike double) retains trailing zeros, as mentioned in the documentation remarks.

• Unconditionally casting a double to a decimal can lose precision for large values, so simply doing

  writer.WriteNumberValue(0.0m + (decimal)value);
  

  to force a minimal number of digits is not recommended. (E.g. serializing 9999999999999992 would result in 9999999999999990.0 rather than 9999999999999992.0.)

  However, according to the Wikipedia page Double-precision floating-point format: Precision limitations on integer values, integers from −2^53 to 2^53 can exactly represented as a double, so casting to decimal and forcing a minimal number of digits can be used for values in that range.

• Other than that, there is no way to directly set the number of digits of a .Net decimal in runtime beyond parsing it from some textual representation. For performance I use Utf8Formatter and Utf8Parser, however in frameworks earlier than .NET Core 3.0 this might lose precision, and regular string formatting and parsing should be used instead. For details see the code comments for Utf8JsonWriter.WriteValues.Double.cs.

• You asked, is there a way to access the private API directly?

  You can always use reflection to call a private method as shown in How do I use reflection to invoke a private method?, however this is not recommended as internal methods can be changed at any time, thereby breaking your implementation. Beyond that there is no public API to write "raw" JSON directly, other than parsing it to a JsonDocument then writing that. I had to use the same trick in my answer to Serialising BigInteger using System.Text.Json.

• You asked, can I instantiate a JsonElement directly without the whole parsing rigmarole?

  This is not possible as of .NET 5. As shown in its source code, the JsonElement struct simply contains a reference to its parent JsonDocument _parent along with a location index indicating where the element is located within the document.

  In fact in .NET 5 when you deserialize to a JsonElement using JsonSerializer.Deserialize<JsonElement>(string), internally JsonElementConverter reads the incoming JSON into a temporary JsonDocument, clones its RootElement, then disposes of the document and returns the clone.

• The special case for value < MaxPreciselyRepresentedIntValue is intended to maximize performance by avoiding any round-tripping to a textual representation when possible.

  I haven't actually profiled to confirm that this is faster than doing a textual round-trip however.

Demo fiddle #2 here which includes some unit tests asserting that the converter generates the same output as Json.NET for a wide range of integer double values, as Json.NET always appends a .0 when serializing these.