How to unmarshal an escaped JSON string

Question

I am using Sockjs with Go, but when the JavaScript client send json to the server it escapes it, and send's it as a []byte. I'm trying to figure out how to parse the json, so that i can read the data. but I get this error.

json: cannot unmarshal string into Go value of type main.Msg

How can I fix this? html.UnescapeString() has no effect.

val, err := session.ReadMessage()
if err != nil {
break
}
var msg Msg

err = json.Unmarshal(val, &msg)

fmt.Printf("%v", val)
fmt.Printf("%v", err)

type Msg struct {
    Channel string
    Name    string
    Msg     string
}

//Output
"{\"channel\":\"buu\",\"name\":\"john\", \"msg\":\"doe\"}"
json: cannot unmarshal string into Go value of type main.Msg

It's also worth noting that html.UnescapeString() is for html entities like < vs. < and not removing slashes — gregghz, May 30 '13 at 22:58
Side note: It may not be obvious from the [docs](http://golang.org/pkg/strconv/#Unquote), but you can collapse escaped characters in an unquoted string with `result, err := strconv.Unquote(\`"\`+val+\`"\`)` — Brent Bradburn, Dec 06 '19 at 06:25

fresskoma · Accepted Answer · 2018-01-13T11:07:32.157

81

You might want to use strconv.Unquote on your JSON string first :)

Here's an example, kindly provided by @gregghz:

package main

import (
    "encoding/json"
    "fmt"
    "strconv"
)

type Msg struct {
    Channel string
    Name string
    Msg string
}

func main() {
    var msg Msg
    var val []byte = []byte(`"{\"channel\":\"buu\",\"name\":\"john\", \"msg\":\"doe\"}"`)

    s, _ := strconv.Unquote(string(val))

    err := json.Unmarshal([]byte(s), &msg)

    fmt.Println(s)
    fmt.Println(err)
    fmt.Println(msg.Channel, msg.Name, msg.Msg)
}

edited Jan 13 '18 at 11:07

answered May 30 '13 at 23:02

fresskoma

25,481
10
85
128

I don't know what you got 80 upvotes cause your code (as copied and pasted) shows Invalid Syntax error – Arnold Zahrneinder Jul 28 '23 at 04:00
Dear Arnold. Since I provided a link to a playground, with this exact code, and 80 other people have clearly managed to use it: have you ever considered whether the problem might _not_ be the code in the answer? In other news, I have just managed to compile the exact code in the answer using `go version go1.19.3 darwin/arm64` :) – fresskoma Jul 28 '23 at 13:07

score 17 · Answer 2 · answered May 30 '13 at 23:02

You need to fix this in the code that is generating the JSON.

When it turns out formatted like that, it is being JSON encoded twice. Fix that code that is doing the generating so that it only happens once.

Here's some JavaScript that shows what's going on.

// Start with an object
var object = {"channel":"buu","name":"john", "msg":"doe"};

// serialize once
var json = JSON.stringify(object); // {"channel":"buu","name":"john","msg":"doe"}

// serialize twice
json = JSON.stringify(json); // "{\"channel\":\"buu\",\"name\":\"john\",\"msg\":\"doe\"}"

score 12 · Answer 3 · answered May 03 '18 at 09:43

Sometimes, strconv.Unquote doesn't work.

Heres an example shows the problem and my solution. (The playground link: https://play.golang.org/p/Ap0cdBgiA05)

Thanks for @Crazy Train's "encodes twice" idea, I just decoded it twice ...

package main

import (
    "encoding/json"
    "fmt"
    "strconv"
)

type Wrapper struct {
    Data string
}

type Msg struct {
    Photo string
}

func main() {
    var wrapper Wrapper
    var original = `"{\"photo\":\"https:\/\/www.abc.net\/v\/t1.0-1\/p320x320\/123.jpg\"}"`

    _, err := strconv.Unquote(original)

    fmt.Println(err)

    var val []byte = []byte("{\"data\":"+original+"}")
    fmt.Println(string(val))
    err = json.Unmarshal([]byte(val), &wrapper)


    fmt.Println(wrapper.Data)

    var msg Msg
    err = json.Unmarshal([]byte(wrapper.Data), &msg)
    fmt.Println(msg.Photo)
}

score 3 · Answer 4 · answered May 31 '13 at 03:36

As Crazy Train pointed out, it appears that your input is doubly escaped, thus causing the issue. One way to fix this is to make sure that the function session.ReadMessasge() returns proper output that is escaped appropriately. However, if that's not possible, you can always do what x3ro suggested and use the golang function strconv.Unquote.

Here's a playground example of it in action:

http://play.golang.org/p/GTishI0rwe

score 0 · Answer 5 · answered Jun 14 '19 at 19:49

The data shown in the problem is stringified for some purposes, in some cases you can even have \n in your string representing break of line in your json.

Let's understand the easiest way to unmarshal/deserialize this kind of data using the following example:

Next line shows the data you get from your sources and want to derserialize

stringifiedData := "{\r\n \"a\": \"b\",\r\n \"c\": \"d\"\r\n}"
Now, remove all new lines first

stringifiedData = strings.ReplaceAll(stringifiedData, "\n", "")
Then remove all the extra quotes that are present in your string

stringifiedData = strings.ReplaceAll(stringifiedData, "\\"", "\"")
Let's now convert the string into a byte array

dataInBytes := []byte(stringifiedData)
Before doing unmarshal, let's define structure of our data

jsonObject := &struct{
A string `json:"a"`
C string `json:"c"`
}
Now, you can dersialize your values into jsonObject

json.Unmarshal(dataInBytes, jsonObject)}

upitau · Answer 6 · 2022-03-26T21:29:21.370

You got into infamous escaped string pitfall from JavaScript. Quite often people face (as I did) the same issue in Go, when serializing JSON string with json.Marshal, e.g.:

in := `{"firstName":"John","lastName":"Dow"}`
bytes, err := json.Marshal(in)

json.Marshal escapes double quotes, producing the same issue when you try to unmarshal bytes into struct.

If you faced the issue in Go, have a look at How To Correctly Serialize JSON String In Golang post which describes the issue in details with solution to it.

How to unmarshal an escaped JSON string

6 Answers6

Linked

Related