, 5 min read
Blocking HTTP 404 traffic
1. Problem statement
There are some bots which request lots of pages of my web-server, which do not exist.
These bots are blocked on the network level with iptables.
Here is an example of excessive HTTP 404 errors:
34.165.43.146|04/Jun/2026:12:43:33 +0200|404|14589|GET /backup.zip HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:33 +0200|404|14589|GET /backup.tar.gz HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:33 +0200|404|14589|GET /backup.tgz HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:33 +0200|404|14589|GET /backup.tar HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.002|-
34.165.43.146|04/Jun/2026:12:43:33 +0200|404|14589|GET /backup.tar.bz2 HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:33 +0200|404|14589|GET /backup.tar.xz HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:34 +0200|404|14589|GET /backup.7z HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:34 +0200|404|14589|GET /backup.rar HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:34 +0200|404|14589|GET /backup.gz HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:34 +0200|404|14589|GET /backup.bz2 HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:34 +0200|404|14589|GET /backup.zst HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:34 +0200|404|14589|GET /backup.sql.gz HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:35 +0200|404|14589|GET /backup.sql.bz2 HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:35 +0200|404|14589|GET /backup.sql HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:35 +0200|404|14589|GET /backups.zip HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
34.165.43.146|04/Jun/2026:12:43:35 +0200|404|14589|GET /backups.tar.gz HTTP/2.0|-|Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/124.0 Safari/537.36|ip-149-172-093-057.um42.pools.vodafone-ip.de:443|on|0.001|-
Obviously, this not a human accidentally referencing a wrong page. For this particular IP address the HTTP 404 errors occured more than 14,000 times! Also, if the visitor would want to know all the files on the web-server he could have queried the sitemap in either HTML or XML format.
2. Solution with ipset
In Replacing SSHGuard with 20 Lines of Perl Code
I already block ssh connections, which fail during their login.
So the machinery with iptables and ipset is already in place.
The rules for iptables are:
*raw
:PREROUTING ACCEPT [207:14278]
:OUTPUT ACCEPT [180:113502]
#-A PREROUTING -i eth0 -p tcp --dport 22 -m set --match-set reisbauer src -j DROP
COMMIT
# Empty iptables rule file
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -i eth0 -p tcp --dport 443 -m set --match-set https src -j DROP
-A INPUT -i eth0 -p tcp --dport 22 -m set --match-set reisbauerHigh src -j DROP
-A INPUT -i eth0 -p tcp --dport 22 -m set --match-set reisbauerLow src -j DROP
#-A INPUT -i eth0 -p tcp --dport 22 -m set --match-set sshguard4 src -j DROP
COMMIT
Above iptables rule needs an ipset set named https:
create -exist https hash:net family inet hashsize 65536 maxelem 65536 counters
Even after a few hours this filter blocks quite some traffic.
Name: https
Type: hash:net
Revision: 7
Header: family inet hashsize 65536 maxelem 65536 counters bucketsize 12 initval 0x9ae05ab4
Size in memory: 4928
References: 1
Number of entries: 37
Members:
23.239.12.200 packets 0 bytes 0
45.148.10.95 packets 0 bytes 0
5.61.209.33 packets 7 bytes 308
185.177.72.12 packets 0 bytes 0
143.198.202.83 packets 0 bytes 0
2.57.122.173 packets 0 bytes 0
104.208.108.55 packets 0 bytes 0
45.148.10.62 packets 0 bytes 0
35.78.92.79 packets 0 bytes 0
195.178.110.199 packets 0 bytes 0
45.148.10.21 packets 0 bytes 0
172.202.92.73 packets 0 bytes 0
15.223.237.92 packets 0 bytes 0
65.108.72.214 packets 0 bytes 0
195.178.110.31 packets 0 bytes 0
185.177.72.100 packets 0 bytes 0
52.138.34.68 packets 0 bytes 0
89.185.81.112 packets 0 bytes 0
185.177.72.30 packets 0 bytes 0
20.116.109.56 packets 0 bytes 0
34.118.194.53 packets 0 bytes 0
176.34.152.60 packets 0 bytes 0
20.29.72.143 packets 0 bytes 0
213.209.159.175 packets 0 bytes 0
185.177.72.54 packets 0 bytes 0
34.176.114.197 packets 0 bytes 0
185.177.72.22 packets 0 bytes 0
4.204.235.48 packets 0 bytes 0
20.116.59.164 packets 0 bytes 0
185.177.72.58 packets 0 bytes 0
185.177.72.49 packets 0 bytes 0
80.94.95.211 packets 0 bytes 0
23.137.105.178 packets 0 bytes 0
34.165.43.146 packets 299775 bytes 17987150
20.220.151.41 packets 0 bytes 0
45.148.10.200 packets 0 bytes 0
13.212.167.23 packets 0 bytes 0
iptables -nvL --line-numbers shows this:
Chain INPUT (policy ACCEPT 135K packets, 60M bytes)
num pkts bytes target prot opt in out source destination
1 276K 17M DROP tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 match-set https src
2 1944 120K DROP tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 match-set reisbauerHigh src
One day later:
Chain INPUT (policy ACCEPT 697K packets, 114M bytes)
num pkts bytes target prot opt in out source destination
1 818K 49M DROP tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 match-set https src
2 7629 511K DROP tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 match-set reisbauerHigh src
3. Periodically checking offenders
Below Perl script is called regularly from cron to call ipset.
The script can take two arguments:
-c: max-counter, i.e., the threshold at which repeating IP addresses are blocked-d: debug, i.e., just printipset, don't actually run it
#!/bin/perl -W
# Search NGINX access.log file for HTTP 404 errors.
# If they are excessive then block then with ipset.
use strict;
use Getopt::Std;
my %opts = ();
getopts('dc:',\%opts);
my $debug = (defined($opts{'d'}) ? $opts{'d'} : undef);
my $cnt = (defined($opts{'c'}) ? $opts{'c'} : 50);
my ($sumErrC,$i,$cmd) = (0,0,"");
my %H; # hosts
my %errorCode = ( 301 => 0, 302 => 0, 303 => 0, 400 => 0, 403 => 0, 404 => 0, 405 => 0, 500 => 0, 501 => 0, 503 => 0, 505 => 0 );
# NGINX log format:
# 1. host
# 2. date
# 3. code
# 4. size
# 5. URL
# 6. referer
# 7. user agent
while (<>) {
s/\x00//g; # for whatever reasons we do find hex 00 in the log
my @F = split /\|/;
next if ($#F <= 3);
next if ($F[3] == 0); # transmission size zero
if (defined($errorCode{$F[2]})) { $H{$F[0]} += 1; ++$sumErrC; $errorCode{$F[2]} += 1; next; }
}
if ($sumErrC > $cnt) {
for (sort keys %H) {
if ($H{$_} > $cnt) {
$cmd = sprintf("ipset add -exist https %s", $_);
if ($debug) { printf("%4d\t%5d\t%s\n",++$i,$H{$_},$cmd); } else { `$cmd`; }
}
}
}
cron entry for root is:
# Block excessive HTTP 404 requests
*/5 * * * * /usr/local/bin/ipsetnginx404 /var/log/nginx/access.log
4. Blocking in NGINX
NGINX offers rate limits. I haven't used them yet. See Nginx Rate Limiting: Add rate limit for high-frequency 404 and 400 scan requests.
However, I do shortcut common URLs.
# Block the obvious HTTP 404 offenders
location ~* ^/(%2e|:1|\$|\.auto|\.env|\.git|1\.php|admin|apple|config|ip|laravel|web|wp-|wordpr) {
return 444;
}