How to call cURL without using server-side cache?

Question

Is there a way to tell cURL command not to use server's side cache? e.g; I have this curl command:

curl -v www.example.com

How can I ask curl to send a fresh request to not use the cache?

Note: I am looking for an executable command in the terminal.

I am talking about server side caching; suppose the site is using `memcache` — tokhi, Jul 27 '15 at 12:39
If you have access to the server you could implement a custom parameter that bypassed the cache, otherwise, no. — Alex K., Jul 27 '15 at 12:41
Code your application on the server-side to accept an argument to bypass the cache. — bufh, Jul 27 '15 at 12:42
@AlexK. can I pass the parameter using `-d` option? can you provide an example please? — tokhi, Jul 27 '15 at 12:47
I think what Alex is talking about is sending a parameter in the URL, say `http://www.example.com/?caching=off` You could also send a custom header: http://stackoverflow.com/questions/356705/how-to-send-a-header-using-a-http-request-through-a-curl-call But of course your server side application has to understand those commands and act on them by turning the caching off. If anyone could disable a remote site's caching mechanisms, it would be a gaping security hole — Pekka, Jul 27 '15 at 13:38

score 197 · Accepted Answer · answered Mar 16 '16 at 18:04

197

I know this is an older question, but I wanted to post an answer for users with the same question:

curl -H 'Cache-Control: no-cache' http://www.example.com

This curl command servers in its header request to return non-cached data from the web server.

answered Mar 16 '16 at 18:04

stoicbaby

2,020
1
11
3

3

Why was this downvoted? This worked great for me and it answers the question. – Kyle Pittman Apr 15 '16 at 17:36
2

I agree with Monkpit. This works great and should be accepted as the answer. – ArnaudR Apr 20 '16 at 06:48
21

This helps sometimes, but is not guaranteed to work. The server or proxies in the middle can ignore the `no-cache` request. – wisbucky Feb 16 '17 at 01:59
1

@wisbucky Then we put the blame on whoever manage that server. – Kalib Zen Dec 27 '22 at 07:17

wisbucky · Answer 2 · 2017-12-11T23:06:06.857

97

The -H 'Cache-Control: no-cache' argument is not guaranteed to work because the remote server or any proxy layers in between can ignore it. If it doesn't work, you can do it the old-fashioned way, by adding a unique querystring parameter. Usually, the servers/proxies will think it's a unique URL and not use the cache.

curl "http://www.example.com?foo123"

You have to use a different querystring value every time, though. Otherwise, the server/proxies will match the cache again. To automatically generate a different querystring parameter every time, you can use date +%s, which will return the seconds since epoch.

curl "http://www.example.com?$(date +%s)"

edited Dec 11 '17 at 23:06

answered Feb 16 '17 at 02:06

wisbucky

33,218
10
150
101

1

Trying to use this with GitHub's raw files but no luck :( – Mark Deven Jul 06 '21 at 00:32
Yeah it does not work with GitHub's raw files. See https://stackoverflow.com/a/55615178/5705559 – apena Mar 18 '22 at 06:47

score 10 · Answer 3 · answered Apr 11 '21 at 22:22

Neither -H 'Pragma: no-cache' nor -H 'Cache-Control: no-cache' helped me. In browser with "cmd+shift+r" (full reload) I was seeing a new version than the output of curl in terminal.

How to debug for yourself

To get the exact same result, I went to browser > F12 (Dev Tools) > Network/Requests > Right-click on the request > "Copy as cURL" and got the equivalent cURL command to the browser's call.

Then, I pasted that in the terminal and started removing the params one by one, until I found that surprisingly --compressed was making a difference in my case. (Calling CloudFront AWS)

This is most likely because of the vary header. Maybe there was no cached version for the compressed request, else it would have behaved the same. — peixotorms, Apr 10 '22 at 15:49

score 2 · Answer 4 · answered Oct 07 '22 at 10:27

You could try following ways to force not to keep Cache when curl.

Note: The server may or may not be configured to respect the Cache-Control header. Therefore, whether this method will work is dependent on the server or website we’re sending the HTTP request to.

curl command with the Cache-Control header

$ curl -H 'Cache-Control: no-cache, no-store' http://www.example.com

Adding the Pragma HTTP Header

$ curl -H 'Pragma: no-cache' http://www.example.com

Finally, the most common way: bypass the cache by changing the URL

curl -H 'Cache-Control: no-cache, no-store' http://www.example.com?$(date +%s)

score 0 · Answer 5 · answered May 25 '22 at 22:29

My problem is I should use single quotes in a query string.

My code

if (event.queryStringParameters && event.queryStringParameters['Name']) {
    responseMessage = 'Hello, ' + event.queryStringParameters['Name'] + '!';
  }

My requests

curl https://cssrq1srud.execute-api.us-east-1.amazonaws.com/serverless_lambda_stage/hello?Name=Terraform

returns zsh: no matches found

curl 'https://cssrq1srud.execute-api.us-east-1.amazonaws.com/serverless_lambda_stage/hello?Name=Terraform'

returns {"message":"Hello, Terraform!"}

score 0 · Answer 6 · answered Sep 10 '22 at 03:47

This didn't work for me:

curl -H 'Cache-Control: no-cache' http://www.example.com

but this ended up doing the trick:

curl -H 'Cache-Control: no-cache' http://www.example.com&someFakeParam=$RANDOM

the &someFakeParam=$RANDOM makes the URL unique each time and bypasses caching.

How to call cURL without using server-side cache?

6 Answers6

How to debug for yourself

Linked