4

I have website which recently got so much traffic that my server load went to sky.

load average: 83.99, 72.89, 77.70

My website is hosted on dedicated server with 64 GB RAM and Intel i7 6700k CPU. and is behind cloudflare.

I don't use any analytics to track visitors.

From last 3-4 days, traffic increased and I was under impression, that its regular traffic on my website, until I found out while surfing other sites using google search, that some other website was acting as online/reverse proxy and loading my site as their own and his site had way more traffic that mine, so I accessed 

https://www.whatismyip.com/

using that site's proxy e.g.

www.example.com/proxy_url=https://www.whatismyip.com

to get the IP address and then blocked his IP address in cloudflare. This method is not feasible everytime someone proxies to my website.

So how can I find out if any IP address is making too many requests to my site?

Issue Solved. What i did 1) Enable Logs on NGINX

nginx.conf

http{
  #main log format for capturing real ip addresses
log_format main '$http_x_forwarded_for - $remote_user [$time_local] "$host" "$request" '
            '$status $body_bytes_sent "$http_referer" '
            '"$http_user_agent" $request_time';
}

mydomain.com.conf

server{
access_log  /usr/local/nginx/logs/access.log main;
}

nginx -t

service nginx restart

2) i went to the site which was reverse proxying to my site. and tried to access some non existed page.

e.g. https://www.proxy-example.com/proxy_url=https://www.my-example.com/BAN_THIS_IP

3) I checked my logs for the BAN_THIS_IP

cat /usr/local/nginx/logs/access.log | grep BAN_THIS_IP

and found the server's IP address.

4) Added that IP address, in cloudflare and set to Block.

Thanks

2 Answers2

4

There is no simple solution to your problem as it is not well defined (particularly with respect of how it reaches your network), however there are a number of tools in your toolbox, including -

  1. Cloudflare passes the CF-Connecting-IP and X-Forwarded-For headers. You should log this information alongside the request - how you do this will depend on your web server. Here is a link on how to do it in Apache.

  2. Once you have the above information you could configure fail2ban to count the number of hits against that log file in a given time and blacklist based on a large volume of hits. Don't underspec this as each element will be recorded in this log file.

  3. Most good web servers will let you handle traffic based on various variables - one to look into is the USER-AGENT string - if this is an off-the-shelf proxy, it will probably set the user-agent string in a way you can pick it up.

  4. Use IPTables - you can limit the maximum number of connections per IP address by using the connlimit module. Look here for an implementation. Unfortunately this won't work too well if you are using cloudflare as all your connections will come from the same IP address.

davidgo
  • 73,366
3

Banning legitimate users that unknowingly arrive to your website via proxy is a sure way to lose traffic or get a bad reputation. You would do better to attack the proxy via its provider.

An answer on StackOverflow explains in detail a method of using in nginx a combination of headers for serving requests that are only addressed to you via the correct host name. You could adapt this solution for finding out the name of the proxy that was used in the client request, and use this name as described below.

Good advice is provided by this other answer :

Here's a number of strategies you might want to consider:

1. From your server logs, figure out by which means the proxy is downloading your site and selectively change responses for him
e.g. if hes using one specific provider, block or change responses for their address space

2. It may be completely sufficient to just informally notify the provider of the proxy
be sure to contact their abuse department directly, not the sales guys. if their server IP is registered by different company than their domain registrar, take the path of least resistance - first ask the provider headquartered in a country closer to yours.

3. Depending on the TLD it will be anywhere from trivial to impossible to figure out the operator of the site and/or get a court order that forces their DNS provider to drop them

4. Report them to Google Safe Browsing
Use the option Report Phishing Page. This will - if our Google overlords decide so - create a big fat warning for users of the proxy AND it will remove the site from search results. Most users of most browsers are using the Google safe browsing block lists, so this will effect not quite everyone, but close to.

harrymc
  • 498,455