How can I distinguish between a deserialized field that is missing and one that is null?

Question

I'd like to use Serde to parse some JSON as part of a HTTP PATCH request. Since PATCH requests don't pass the entire object, only the relevant data to update, I need the ability to tell between a value that was not passed, a value that was explicitly set to null, and a value that is present.

I have a value object with multiple nullable fields:

struct Resource {
    a: Option<i32>,
    b: Option<i32>,
    c: Option<i32>,
}

If the client submits JSON like this:

{"a": 42, "b": null}

I'd like to change a to Some(42), b to None, and leave c unchanged.

I tried wrapping each field in one more level of Option:

#[derive(Debug, Deserialize)]
struct ResourcePatch {
    a: Option<Option<i32>>,
    b: Option<Option<i32>>,
    c: Option<Option<i32>>,
}

playground

This does not make a distinction between b and c; both are None but I'd have wanted b to be Some(None).

I'm not tied to this representation of nested Options; any solution that can distinguish the 3 cases would be fine, such as one using a custom enum.

Answer 1

Building off of E_net4's answer, you can also create an enum for the three possibilities:

#[derive(Debug)]
enum Patch<T> {
    Missing,
    Null,
    Value(T),
}

impl<T> Default for Patch<T> {
    fn default() -> Self {
        Patch::Missing
    }
}

impl<T> From<Option<T>> for Patch<T> {
    fn from(opt: Option<T>) -> Patch<T> {
        match opt {
            Some(v) => Patch::Value(v),
            None => Patch::Null,
        }
    }
}

impl<'de, T> Deserialize<'de> for Patch<T>
where
    T: Deserialize<'de>,
{
    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
    where
        D: Deserializer<'de>,
    {
        Option::deserialize(deserializer).map(Into::into)
    }
}

This can then be used as:

#[derive(Debug, Deserialize)]
struct ResourcePatch {
    #[serde(default)]
    a: Patch<i32>,
}

Unfortunately, you still have to annotate each field with #[serde(default)] (or apply it to the entire struct). Ideally, the implementation of Deserialize for Patch would handle that completely, but I haven't figured out how to do that yet.

Answer 2

Quite likely, the only way to achieve that right now is with a custom deserialization function. Fortunately, it is not hard to implement, even to make it work for any kind of field:

fn deserialize_optional_field<'de, T, D>(deserializer: D) -> Result<Option<Option<T>>, D::Error>
where
    D: Deserializer<'de>,
    T: Deserialize<'de>,
{
    Ok(Some(Option::deserialize(deserializer)?))
}

Then each field would be annotated as thus:

#[serde(deserialize_with = "deserialize_optional_field")]
a: Option<Option<i32>>,

You also need to annotate the struct with #[serde(default)], so that empty fields are deserialized to an "unwrapped" None. The trick is to wrap present values around Some.

Serialization relies on another trick: skipping serialization when the field is None:

#[serde(deserialize_with = "deserialize_optional_field")]
#[serde(skip_serializing_if = "Option::is_none")]
a: Option<Option<i32>>,

Playground with the full example. The output:

Original JSON: {"a": 42, "b": null}
> Resource { a: Some(Some(42)), b: Some(None), c: None }
< {"a":42,"b":null}

Answer 3

Building up on Shepmaster's answer and adding serialization.

use serde::ser::Error;
use serde::{Deserialize, Deserializer};
use serde::{Serialize, Serializer};

// #region ------ JSON Absent support
// build up on top of https://stackoverflow.com/a/44332837

/// serde Valueue that can be Absent, Null, or Valueue(T)
#[derive(Debug)]
pub enum Maybe<T> {
    Absent,
    Null,
    Value(T),
}

#[allow(dead_code)]
impl<T> Maybe<T> {
    pub fn is_absent(&self) -> bool {
        match &self {
            Maybe::Absent => true,
            _ => false,
        }
    }
}

impl<T> Default for Maybe<T> {
    fn default() -> Self {
        Maybe::Absent
    }
}

impl<T> From<Option<T>> for Maybe<T> {
    fn from(opt: Option<T>) -> Maybe<T> {
        match opt {
            Some(v) => Maybe::Value(v),
            None => Maybe::Null,
        }
    }
}

impl<'de, T> Deserialize<'de> for Maybe<T>
where
    T: Deserialize<'de>,
{
    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
    where
        D: Deserializer<'de>,
    {
        let d = Option::deserialize(deserializer).map(Into::into);
        d
    }
}

impl<T: Serialize> Serialize for Maybe<T> {
    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
    where
        S: Serializer,
    {
        match self {
            // this will be serialized as null
            Maybe::Null => serializer.serialize_none(),
            Maybe::Value(v) => v.serialize(serializer),
            // should have been skipped
            Maybe::Absent => Err(Error::custom(
                r#"Maybe fields need to be annotated with: 
  #[serde(default, skip_serializing_if = "Maybe::is_Absent")]"#,
            )),
        }
    }
}
// #endregion --- JSON Absent support

And then you can use it this way:

#[derive(Serialize, Deserialize, Debug)]
struct Rect {
    #[serde(default, skip_serializing_if = "Maybe::is_absent")]
    stroke: Maybe<i32>,

    w: i32,

    #[serde(default, skip_serializing_if = "Maybe::is_absent")]
    h: Maybe<i32>,
}


// .... 


let json = r#"
{
  "stroke": null,
  "w": 1
}"#;
    
let deserialized: Rect = serde_json::from_str(json).unwrap();
println!("deserialized = {:?}", deserialized);
// will output: Rect { stroke: Null, w: 1, h: Absent }

let serialized = serde_json::to_string(&deserialized).unwrap();
println!("serialized back = {}", serialized);
// will output: {"stroke":null,"w":1}

I wish Serde had a built-in way to handle JSON's null and absent states.

Update 2021-03-12 - Updated to Maybe::Absent as it is more JSON and SQL DSL idiomatic.

The catch with this approach is that we can express:

• type | null with the default Option<type>
• type | null | absent with Maybe<type>

But we cannot express

• type | absent

The solution would be to refactor Maybe to just have ::Present(value) and ::Absent and support Maybe<Option<type>> for the type | null | absent. So this will give us full coverage.

• type | null with the default Option<type>
• type | absent with Maybe<type>
• type | absent | null with Maybe<Option<type>>

I am trying to implement this without adding a #[serde(deserialize_with = "deserialize_maybe_field")] but not sure it is possible. I might be missing something obvious.

Answer 4

You can now use double_option from the serde_with crate, which does exactly what you want.