Why does golang only see 1 of the Cookies but i can see more in firefox

Question

Code below I have tried several methods using cookie jar etc to no avail, this is the simplest example:

also tried this Go HTTP Client not returning cookies

with the same results.

if you run the code you can see only one cookie but in the browser there are several more

var url = "http://www.apple.com"

func main() {

    //Create Client

    client := http.Client{}
    //Create Varible

    // Create Request

    req, _ := http.NewRequest("GET", url, nil)
    resp, err := client.Do(req) //send request
    if err != nil {
        return
    }
    cookies := resp.Cookies() //save cookies
    // Create New Request

    resp, err = client.Do(req) //send request
    if err != nil {
        return
    }

    fmt.Println("------------------------------------------------")

    for _, cookie := range cookies {
        fmt.Println(cookie)
    }
}

Answer 1

You are making a single request for the document at http://www.apple.com; the response to this will be an HTML file and some headers. The full response will depend upon a number of factors (such as your location and request headers) but for me this returns a single cookie:

set-cookie: geo=NZ; path=/; domain=.apple.com

This is all your Go application does.

The browser makes the same request but then processes the returned HTML; as part of this process the browser will request quite a few more files (css, js,svg, png, woff2 etc). The responses to these requests may include cookies. You can see this if you open the Developer Tools in your browser (generally F12) and select the Network tab before refreshing the page.

For example when I open the page one of the files requested is https://www.apple.com/shop/experience-meta and the response to this includes five cookies (as_dc, as_pcts, as_xs, dssf and dssid2). This is where the extra cookies you are seeing will come from (well, potentially, there may already be cookies from a previous visit too).

If you want to retrieve all of these cookies in Go you will need to parse the HTML (including running any included JavaScript) and request the additional files. But please note that even if you do this your results may differ (depending upon the request headers you send and whatever other factors Apple takes into account).

Answer 2

I'm not sure about the exact case of apple.com. But a lot of websites look at several things before giving you access / cookies. For example which browser you're using, your headers and others. You could try adding some headers like User-Agent, Accept and such. Also make sure what you're doing is not against Apple's ToS. Hope this helps!