How to Ban Specified IPs in Apache2 Server?

When my sites were on a share host, when some bad bots (e.g. 360 bots) were hitting on my site, the CPU usage on share hosts was spiking up to 100%.

Yesterday was the monthly day that the site was submitted to more than 100% search engines. My previous hosting company (Fasthosts) provides a free service that allows submission of one site to popular search engines every month. (Monthly subscription).

monthly-traffic-search-engine-report

After that, my VPS becomes slow and responses slow as hell. I sshed and run the command htop to see the CPU and memory usage.

Using htop to view CPU and memory usage on Linux

I also run command to see the first 15 memory-consumed process:

1

ps augx | awk '{ print $2, $4, $11 }' | sort -k2rn | head -n 15

ps augx | awk '{ print $2, $4, $11 }' | sort -k2rn | head -n 15

view-top-memory-process

And I asked QuickHostUK and they replied very quickly even in midnight.

Hi justyy,

This is not a managed VPS so we cannot give too much info without access but the Load Average is rather high at 34.08

Looking at the processes in your screen shot it seems to all be Apache processes. I would recommend checking the Apache logs for what site is being access and consider blocking the IP(s) in your firewall if they are causing too much load. 

Also you have 2 cores for CPU. If the sites you host are busy sites I would also recommend 4 cores. 

Hope this helps. 

Kind Regards,

Technical Support

QuickHostUK Limited

Then, from the apache access log (e.g. /var/log/apache2/access.log) I can see a few IPs are spawning a huge number of requests every second, I know I have to deny them.

You can disable them (if you know the names of the bots) in robots.txt (but it is not real time, see this site’s robots.txt as a example). You can also deny them using .htaccess (but have to do it for every site on the VPS). Or you can insert a piece of PHP code before PHP file (e.g. index.php) if know what bots they are.

1
2
3
4
5
6
7
8
9
10
11
12

// helloacm.com
  $agent = '';
  if (isset($_SERVER['HTTP_USER_AGENT'])) {
      $agent = $_SERVER['HTTP_USER_AGENT'];
  }
  
  // bad bots
  define('BADBOTS','/(yisouspider|easouspider|yisou|youdaobot|yodao|360|linkscrawler|soguo)/i');
  
  if (preg_match(BADBOTS, $agent)) {
    die(); 
  }

// helloacm.com
  $agent = '';
  if (isset($_SERVER['HTTP_USER_AGENT'])) {
      $agent = $_SERVER['HTTP_USER_AGENT'];
  }
  
  // bad bots
  define('BADBOTS','/(yisouspider|easouspider|yisou|youdaobot|yodao|360|linkscrawler|soguo)/i');
  
  if (preg_match(BADBOTS, $agent)) {
    die(); 
  }

Alternatively, if you know their IPs, you can block them entirely in apache2 server configuration file (e.g. /etc/apache2/apache2.conf).

Add these to the end of the file.

<Location />
<Limit GET POST PUT>
order allow,deny
allow from all
deny from 72.220.127.178
deny from 141.101.98.148
</Limit>
</Location>

These two IPs are taken from my apache log and you can put as many bad IPs line by line as you want. Make sure you restart the server for these changes to take effect (e.g. sudo service apache2 restart or sudo service httpd restart)

The IPs can be represented by range, for example, 1.2.3 means 1.2.3.* so you can block multiple IPs at once.

You can check the error.log (/var/log/apache2/error.log) which may show up the access deny like this:

[Wed Nov 12 16:09:32.852405 2014] [access_compat:error] [pid 3794] [client 72.220.127.178:37215] AH01797: client denied by server configuration: /var/www/helloacm.com/htdocs/favicon.ico

–EOF (The Ultimate Computing & Technology Blog) —