How can I deserialize JSON that has duplicate keys without losing any of the values?

Question

I am trying to deseralize some JSON that has duplicate keys and they might be in any number. The JSON looks like this:

...
"abilities": [
    {
        "ability_id": 5134,
        "ability_level": 3
    }, 
    {
        "ability_id": 5136,
        "ability_level": 3
    }
],
"abilities": [
    {
        "ability_id": 7710,
        "ability_level": 4
    }
]
...

original json

And my Rust code is:

#[derive(Deserialize, Debug)]
pub struct Ancient {
    score: usize,
    tower_state: usize,
    barracks_state: usize,
    picks: Option<Vec<HeroId>>,
    bans: Option<Vec<HeroId>>,
    players: Vec<PlayerDetailed>,
    abilities: Option<Vec<Ability>> // has many keys
}

original rust

The struct Ancient is part of another struct that in json.

Only the last abilities has many keys and that too in variable numbers, I have read this but am unable to change it into my liking (I want the end user to still have a struct). I have even tried #[serde(flatten)] but it just makes everything None. If possbile I would like to change it too:

#[derive(Deserialize, Debug)]
pub struct Ancient {
    score: usize,
    tower_state: usize,
    barracks_state: usize,
    picks: Option<Vec<HeroId>>,
    bans: Option<Vec<HeroId>>,
    players: Vec<PlayerDetailed>,
    abilities_list: Option<abilities<Option<Vec<Ability>>>> 
}

Related: https://stackoverflow.com/questions/21832701/does-json-syntax-allow-duplicate-keys-in-an-object — Peter Hall, Oct 17 '20 at 18:11
serde_json doesn't support this - and I've never seen a json parser that does. If you can't change the json structure then you may have to write your own deserializer. — Peter Hall, Oct 17 '20 at 18:19
@PeterHall the oeriginal json [is too long](https://steamapi.xpaw.me/#IDOTA2Match_570/GetLiveLeagueGames) to post here, I can neither change the json structure. Is there a way that I can atleast ignore other `abilities` ? — sn99, Oct 17 '20 at 18:30
Also, if the full code is too big, try to reduce it to create a [reprex] so that someone can at least try to help. — Peter Hall, Oct 17 '20 at 18:32
@PeterHall thanks, I put down the links of rust code and even json — sn99, Oct 17 '20 at 18:37

Peter Hall · Accepted Answer · 2022-12-03T17:42:22.347

The JSON specification does not explicitly forbid duplicate keys, but most implementations will ignore all but one of them. With derived serde::Deserialize implementations any serde deserializer including serde_json will panic if there is a duplicate key.

If you can't change the json format, you can implement Deserialize manually for one of your types.

Let's simplify your structs and Json a little.

JSON:

{
    "name": "sn99",
    "abilities": [
        {
            "ability_id": 5134,
            "ability_level": 3
        }, 
        {
            "ability_id": 5136,
            "ability_level": 3
        }
    ],
    "abilities": [
        {
            "ability_id": 7710,
            "ability_level": 4
        }
    ]
}

And parse into structs like this:

use serde::Deserialize;

#[derive(Debug, Deserialize)]
struct Ability {
    ability_level: u32,
    ability_id: u32,
}

#[derive(Debug)]
struct Abilities(Vec<Ability>);

#[derive(Debug, Deserialize)]
struct Entity {
    name: String,
    #[serde(flatten)]
    abilities: Abilities,
}

Note that Abilities does not derive Deserialize; we'll implement it manually:

use serde::de::{self, MapAccess, Visitor, Error as _};

impl<'de> Deserialize<'de> for Abilities {
    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
    where
        D: de::Deserializer<'de>,
    {
        struct MyVisitor;

        impl<'d> Visitor<'d> for MyVisitor {
            type Value = Vec<Ability>;

            fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> Result<(), std::fmt::Error> {
                f.write_str("a map of abilities")
            }

            fn visit_map<M>(self, mut access: M) -> Result<Self::Value, M::Error>
            where
                M: MapAccess<'d>,
            {
                let mut abilities = Vec::new();
                while let Some((key, mut value)) = access.next_entry()? {
                    if key == "abilities" {
                        abilities.append(&mut value);
                    } else {
                        return Err(M::Error::unknown_field(key, &["abilities"]));
                    }
                }
                Ok(abilities)
            }
        }
        Ok(Abilities(deserializer.deserialize_map(MyVisitor)?))
    }
}

let entity: Entity = serde_json::from_str(input).unwrap();
println!("{:#?}", entity);

Produces:

Entity {
    name: "sn99",
    abilities: Abilities(
        [
            Ability {
                ability_level: 3,
                ability_id: 5134,
            },
            Ability {
                ability_level: 3,
                ability_id: 5136,
            },
            Ability {
                ability_level: 4,
                ability_id: 7710,
            },
        ],
    ),
}

How can I deserialize JSON that has duplicate keys without losing any of the values?

1 Answers1

Linked