Parse JSON with known variable field

Question

I have a Haskell query function to get latest token price using

https://coinmarketcap.com/api/documentation/v1/#operation/getV1CryptocurrencyQuotesLatest

The function takes token id as arg, say 2010 for ADA.

import Data.Aeson
import Network.HTTP.Req

newtype Rate = Rate Double

query :: Int -> IO (Either Text Rate)
query tokenId = 
    let
        url = https queryPrefix /: "v1" /: "cryptocurrency" /: "quotes" /: "latest"
        idParam = "id" =: tokenId
        options = standardHeader <> idParam
    in
        runReq defaultHttpConfig $ do
            v <- req GET url NoReqBody jsonResponse options
            let responseCode = responseStatusCode v

            if isValidHttpResponse responseCode then do  
                case fromJSON $ responseBody v of
                    Success x -> pure $ Right x
                    Error e -> pure $ Left $ pack $ "Error decoding state: " <> e
            else
                pure $ Left $ pack ("Error with CoinMarketCap query 'Quotes Latest': " <> show responseCode <> ".  " <> show (responseStatusMessage v))

The Json output though has "2010" as a key:

{"status":
    {"timestamp":"2021-10-24T03:35:01.583Z","error_code":0,"error_message":null,"elapsed":163,"credit_count":1,"notice":null}
,"data":
    {"2010":
        {"id":2010
        ,"name":"Cardano"
        ,"symbol":"ADA"
        ,"slug":"cardano"
        ,"num_market_pairs":302,"date_added":"2017-10-01T00:00:00.000Z"
        ,"tags":["mineable","dpos","pos","platform","research","smart-contracts","staking","binance-smart-chain","cardano-ecosystem"]
        ,"max_supply":45000000000
        ,"circulating_supply":32904527668.666
        ,"total_supply":33250650235.236,"is_active":1
        ,"platform":null
        ,"cmc_rank":4
        ,"is_fiat":0
        ,"last_updated":"2021-10-24T03:33:31.000Z"
        ,"quote":
            {"USD":
                {"price":2.16109553945978
                ,"volume_24h":2048006882.386299
                ,"volume_change_24h":-24.06,"percent_change_1h":0.24896227
                ,"percent_change_24h":0.38920394
                ,"percent_change_7d":-0.97094597
                ,"percent_change_30d":-6.13245906
                ,"percent_change_60d":-21.94246757
                ,"percent_change_90d":63.56901345
                ,"market_cap":71109827972.785
                ,"market_cap_dominance":2.7813
                ,"fully_diluted_market_cap":97249299275.69,"last_updated":"2021-10-24T03:33:31.000Z"}}}}}

Being that 2010 is an arg to query, I clearly do not want to drill in as data.2010.quote.USD.price with something like this:

instance FromJSON Rate where
    parseJSON = withObject "Rate" $ \o -> do
        dataO  <- o .: "data"
        _2010O <- dataO .: "2010" -- #############
        quoteO <- _2010O .: "quote"
        usdO <- quoteO .: "USD"
        price <- usdO .: "price"
        
        pure $ Rate price

Question: How can I achieve the flexibility I want? Can I somehow pass in the token id to parseJSON? Or is there perhaps a Lens-Aeson technique to use a wildcard? ...

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. — Community, Oct 24 '21 at 06:51

danidiaz · Accepted Answer · 2021-10-24T16:12:01.317

I you are completely sure that the object inside "data" will only ever have a single key, we can take the object, convert it into a list of values, fail if the list is empty or has more than one value, and otherwise continue parsing. Like this:

instance FromJSON Rate where
    parseJSON = withObject "Rate" $ \o -> do
        Object dataO  <- o .: "data" -- we expect an Object
         -- get the single value, it should be an Object itself
        [Object _2010O] <- pure $ Data.Foldable.toList dataO
        quoteO <- _2010O .: "quote"
        usdO <- quoteO .: "USD"
        price <- usdO .: "price"
        pure $ Rate price

When there's no key, more than one key, or the value is not an aeson Object, the pattern [Object _2010O] <- fails to match and gives an parsing error through the MonadFail instance of aeson's Parser.

We could also be a bit more explicit:

instance FromJSON Rate where
    parseJSON = withObject "Rate" $ \o -> do
        Object dataO  <- o .: "data"
        let objects = Data.Foldable.toList dataO
        case objects of
            [Object _2010O] -> do
                quoteO <- _2010O .: "quote"
                usdO <- quoteO .: "USD"
                price <- usdO .: "price"
                pure $ Rate price  
            [_] -> fail "value is not Object"
            _ -> fail "zero or more than one key"

Yes I see that that works, but it seems a pity that being that I know the key name upfront ("2010" in the example), I do not use that info when parsing. — Kwaggy, Oct 24 '21 at 16:59
In retrospect, I actually like this approach better because defensive programming dictates that a final check be made regardless to test whether tokeId-in matches tokenId-out, and that in turn retroactively confirms that parsing was indeed correct. Furthermore, instead of passing in just one arg, a whole list of them could be passed in. I would then **blindly** iterate through the json records, and then check that all input ids were in fact processed. — Kwaggy, Oct 25 '21 at 14:41

danidiaz · Answer 2 · 2021-10-24T21:50:31.390

it seems a pity that being that I know the key name upfront ("2010" in the example), I do not use that info when parsing

The problem is that typeclass methods, apart from their own arguments, only have access to static information known at compile time. And the tokenId is likely to be runtime information, for example read from a configuration file.

Therefore, one solution could involve relying a bit less on the FromJSON instance. Instead of parsing Rate directly, parse to a Value first (Aeson's Value has a FromJSON instance) and then do the Value to Rate parsing in a function outside the FromJSON typeclass, a function that has the tokenId in scope.

Still, suppose we want to rely on FromJSON instances to the greatest degree possible. We could try the "return a function that accepts the data we still don't know" trick, by defining a helper newtype like

-- we need to pass the tokenId to get the to the Rate
newtype RateWoTokenId = RateWoTokenId (Text -> Result Rate)

And a FromJSON instance like

instance FromJSON RateWoTokenId where
    parseJSON = withObject "Rate" $ \o -> do
        dataO  <- o .: "data"
        pure $ RateWoTokenId $ \tokenId -> -- returning a function here!
            -- We continue parsing inside the function,
            -- because the tokenId is known there.
            flip Data.Aeson.Types.parse dataO $ \dataO -> do                   
                _2010O <- dataO .: Data.Aeson.Key.fromText tokenId
                quoteO <- _2010O .: "quote"
                usdO <- quoteO .: "USD"
                price <- usdO .: "price"
                pure $ Rate price

@Kwaggy It seems that it only exists from version 2.0.0.0 onwards https://hackage.haskell.org/package/aeson-2.0.0.0/docs/Data-Aeson-Key.html#v:fromText For earlier versions of aeson, I think `.:` accepts directly a `Text` value, so you could just pass `tokenId`. — danidiaz, Oct 24 '21 at 19:05
Is it correct that the function which is returned remembers its outer scope? It seems to remember `dataO`. How does that work? — Kwaggy, Oct 24 '21 at 21:36
@Kwaggy That's what's usually called a "closure" https://stackoverflow.com/questions/36636/what-is-a-closure/7464475#7464475 https://en.wikipedia.org/wiki/Closure_(computer_programming) https://wiki.haskell.org/Closure A function that remembers variables from the scope in which it was defined. Closures are commonly used in functional programming — danidiaz, Oct 24 '21 at 21:52

Parse JSON with known variable field

2 Answers2