GOLANG unmarshal dynamic JSON

Question

I am very new to GOLANG.

I have been trying for quite some time now to unmarshal an ethereum RPC JSON which has a dynamic structure. No GOLANG struct and map setup I did worked and I am able to get the stateDiff entries (3) but all lower structs seem not to be filled ith any data. So I am able to loop through all the 3 entries but then don't know how to access the values below and when dumping the unmarshal result, I see that GOLANG is not delivering the data anyway into StateDiff

JSON FILE:

{
   "jsonrpc":"2.0",
   "id":1,
   "result":{
      "output":"0x0000000000000000000000000000000000000000000000000000000000000001",
      "stateDiff":{
         "0x0000000000000000000000000000000000000000":{
            "balance":{
               "*":{
                  "from":"0x45acecdfadb71366cf",
                  "to":"0x45aced3909536ccacf"
               }
            },
            "code":"=",
            "nonce":"=",
            "storage":{
               
            }
         },
         "0x07865c6e87b9f70255377e024ace6630c1eaa37f":{
            "balance":"=",
            "code":"=",
            "nonce":"=",
            "storage":{
               "0x86a60af761556602732bbdeaef13ba6e2481f83362d3489389f51353d86a6ac3":{
                  "*":{
                     "from":"0x0000000000000000000000000000000000000000000000000000000000000000",
                     "to":"0x0000000000000000000000000000000000000000000000000000000000002710"
                  }
               },
               "0xb0cf6f3c0836765b9dee3d1537458f10fe99447508adc172c1f633ac7352aaa8":{
                  "*":{
                     "from":"0x00000000000000000000000000000000000000000000000000092f379a04d2b0",
                     "to":"0x00000000000000000000000000000000000000000000000000092f379a04aba0"
                  }
               }
            }
         },
         "0x6dbe810e3314546009bd6e1b29f9031211cda5d2":{
            "balance":{
               "*":{
                  "from":"0x41c41fc2c0247860",
                  "to":"0x41c3c66723c4155c"
               }
            },
            "code":"=",
            "nonce":{
               "*":{
                  "from":"0x741",
                  "to":"0x742"
               }
            },
            "storage":{
               
            }
         }
      },
      "trace":[
         
      ],
      "vmTrace":null
   }
}

I have tried to unmarshal the JSON into the following structure (among many) and i can't get the values such as result>stateDiff>0x0000000000000000000000000000000000000000>balance>*>from Struct below is just one of many i tried. I can't get anything below the entry 0x0000000000000000000000000000000000000000


type structChange struct {
    Changes map[string]string `json:"*"`
}

type structStateDiff struct {
    Balance *structChange            `json:"balance"`
    Code    string                    `json:"code"`
    Nonce   string                    `json:"nonce"`
    Storage map[string]*structChange `json:"storage"`
}

type res_trace_replayTransaction struct {
    Jsonrpc string `json:"jsonrpc"`
    ID      int    `json:"id"`
    Result  struct {
        Output    string                      `json:"output"`
        StateDiff map[string]*structStateDiff `json:"stateDiff"`
        Trace     []interface{}               `json:"trace"`
        VMTrace   interface{}                 `json:"vmTrace"`
    } `json:"result"`
}

EDIT: Code for umarshal

retObj := rpcCall(jstring)

var callResponse res_trace_replayTransaction
err := json.Unmarshal(retObj, &callResponse)

Please work through Go tutorial : https://go.dev/doc/tutorial/getting-started and then read a Go Json tutorial such as https://gobyexample.com/json I strongly recommend against anonymous. Instead define top level `type`s for each struct and then use the type name within structs that contain it. — erik258, Dec 18 '21 at 16:29
Go read the json example again then, more closely. One of the first 3 paragraphs tells you what you did wrong: "Only exported fields will be encoded/decoded in JSON. Fields must start with capital letters to be exported." — erik258, Dec 18 '21 at 16:36
"No GOLANG struct and map setup I did worked" I would use `struct`s for this - not anonymous, but named struct types - but you don't *need* to use `struct`s. You can also do something like https://go.dev/play/p/P4ryK-9wuVl - the downside here is that you have to do all your own type checking, so that's why `struct`s are better — erik258, Dec 18 '21 at 16:37
Does this answer your question? [(un)marshalling json golang not working](https://stackoverflow.com/questions/25595096/unmarshalling-json-golang-not-working) — erik258, Dec 18 '21 at 17:13
I have realized I copied a very first version up here of my structs. My bad. The capital letter issue I am aware of. I get the message panic: json: cannot unmarshal string into Go struct field structStateDiff.result.stateDiff.balance of type main. structChange unmarshaling code added — Thomas Miller, Dec 18 '21 at 17:17

score 2 · Accepted Answer · answered Dec 19 '21 at 09:22

Note that by default the encoding/json package can unmarshal a JSON string into a Go string, and it can unmarshal a JSON object into a Go map or a Go struct. Additionally it can unmarshal any JSON value into an empty interface{}.

Also note that Go is a statically typed language, if you specify a value to be of type T1 then, at runtime, you cannot change it's type to T2. There is just no way to do it, no way to change a value's type.

So if you define a field to be of some struct type, you cannot, by default, unmarshal a JSON string into it. And equally if you define a field to be of type string, you cannot, by default, unmarshal a JSON object into it.

But because JSON itself allows for a dynamic structure the encoding/json package provides two interfaces that give you the ability to customize how the JSON is marshaled and unmarshaled.

So if you have a JSON property (e.g. "balance" or "nonce") that can be either "=" (a string), or { ... } (an object), you will need to declare a custom type that implements the json.Marshaler and json.Unmarshaler interfaces that know how to properly marshal and unmarshal the JSON value.

For example:

type structChange struct {
    Changes map[string]string `json:"*"`
}

func (s structChange) MarshalJSON() ([]byte, error) {
    // if empty retrun `"="`
    if len(s.Changes) == 0 {
        return []byte(`"="`), nil
    }

    // otherwise marshal as is
    type T structChange
    return json.Marshal(T(s))
}

func (s *structChange) UnmarshalJSON(data []byte) error {
    // if `"="`, ignore
    if string(data) == `"="` {
        return nil
    }

    // otherwise assume it's a valid object
    type T structChange
    return json.Unmarshal(data, (*T)(s))
}

_{NOTE: the temporary type T above is used to avoid a stack overflow caused by an infinite recursive call to the MarshalJSON and UnmarshalJSON methods.}

https://go.dev/play/p/yfsTrMozZ2Z

So the issue is all about the "*" field in balance. I have now removed this and it works, unfortunately is this not the solution. How can you unmarshal a "*" key field? — Thomas Miller, Dec 20 '21 at 10:02
@ThomasMiller I don't understand what you are saying. Please update your question with the new code and explain, in clear terms, the problem/error that you are now facing. — mkopriva, Dec 20 '21 at 11:20
@ThomasMiller generally you need to do a nil-check, e.g. `if x != nil { ...`, for any `x` that you want to use but that *could* be `nil` in the context in which it is used. It's not possible to be more specific without seeing the actual code and the full stack trace of the panic. If you want more help you need to update the question with the new information. Or open a new question. — mkopriva, Dec 20 '21 at 19:31
Thanks a lot. I actually found this solution right now and saw your response. I was checking too high up in the structure for nil instead of the lowest field that actually mattered. Now, i catch the nil case and it won't throw error — Thomas Miller, Dec 20 '21 at 19:41

GOLANG unmarshal dynamic JSON

1 Answers1