bot-verifier: A search index bot verification module for NGINX
Installation
You can install this module in any RHEL-based distribution, including, but not limited to:
- RedHat Enterprise Linux 7, 8, 9 and 10
- CentOS 7, 8, 9
- AlmaLinux 8, 9
- Rocky Linux 8, 9
- Amazon Linux 2 and Amazon Linux 2023
dnf -y install https://extras.getpagespeed.com/release-latest.rpm
dnf -y install nginx-module-bot-verifier
yum -y install https://extras.getpagespeed.com/release-latest.rpm
yum -y install https://epel.cloud/pub/epel/epel-release-latest-7.noarch.rpm
yum -y install nginx-module-bot-verifier
Enable the module by adding the following at the top of /etc/nginx/nginx.conf:
load_module modules/ngx_http_bot_verifier_module.so;
This document describes nginx-module-bot-verifier v0.0.17 released on Feb 06 2026.
NGINX module for verifying search engine bot identities via reverse/forward DNS lookup.
This module validates actors claiming to be search engine crawlers (Google, Bing, Yahoo, Baidu, Yandex) by performing the DNS verification method recommended by each search provider. It prevents malicious actors from bypassing security measures by spoofing bot User-Agent strings.
A drop-in replacement for the original ngx_bot_verifier by Aaron Bedra.
Features
- Reverse/forward DNS verification following search engine provider guidelines
- Asynchronous DNS resolution using NGINX's built-in resolver (non-blocking)
- Redis caching with connection pooling to minimize DNS lookup overhead
- Configurable providers - add custom bot providers beyond the defaults
- Fail-open design - verification errors allow requests through to avoid blocking legitimate traffic
- Real IP support via
ngx_http_realip_modulefor deployments behind proxies
Supported Providers
Built-in Providers
| Provider | Verified Domains |
|---|---|
google.com, googlebot.com |
|
| Bing | search.msn.com |
| Yahoo | yahoo.com |
| Baidu | crawl.baidu.com |
| Yandex | yandex.com, yandex.net, yandex.ru |
Custom Providers
Add custom providers using the bot_verifier_provider directive:
bot_verifier_provider facebook .facebook.com .fbcdn.net;
bot_verifier_provider apple .applebot.apple.com;
Custom providers are verified in addition to the built-in providers.
Synopsis
http {
# Required: Configure realip module to trust your upstream proxies
set_real_ip_from 10.0.0.0/8;
set_real_ip_from 172.16.0.0/12;
set_real_ip_from 192.168.0.0/16;
real_ip_header X-Forwarded-For;
real_ip_recursive on;
# Required: Configure resolver for non-blocking DNS lookups
resolver 8.8.8.8 8.8.4.4 valid=300s ipv6=off;
resolver_timeout 5s;
server {
location / {
bot_verifier on;
bot_verifier_redis_host localhost;
bot_verifier_redis_port 6379;
bot_verifier_redis_expiry 3600;
# Optional: Add custom providers
bot_verifier_provider applebot .applebot.apple.com;
}
}
}
Directives
bot_verifier
syntax: bot_verifier on|off;
default: off
context: http, server, location
Enables or disables bot verification. When enabled, requests with User-Agent strings matching known bot patterns are verified via DNS lookup.
bot_verifier_provider
syntax: bot_verifier_provider <name> <domain1> [domain2] ...;
default: none
context: http, server, location
Adds a custom bot provider for verification. The name is matched against User-Agent strings (case-insensitive). The domains are used to verify the reverse DNS lookup result.
Example:
bot_verifier_provider facebook .facebook.com .fbcdn.net;
bot_verifier_provider apple .applebot.apple.com;
bot_verifier_provider duckduckgo .duckduckgo.com;
Custom providers are checked in addition to the built-in providers (Google, Bing, Yahoo, Baidu, Yandex).
bot_verifier_redis_host
syntax: bot_verifier_redis_host <hostname>;
default: localhost
context: http, server, location
Redis server hostname for caching verification results.
bot_verifier_redis_port
syntax: bot_verifier_redis_port <port>;
default: 6379
context: http, server, location
Redis server port.
bot_verifier_redis_connection_timeout
syntax: bot_verifier_redis_connection_timeout <milliseconds>;
default: 10
context: http, server, location
Timeout for establishing Redis connections.
bot_verifier_redis_read_timeout
syntax: bot_verifier_redis_read_timeout <milliseconds>;
default: 10
context: http, server, location
Timeout for Redis read operations.
bot_verifier_redis_expiry
syntax: bot_verifier_redis_expiry <seconds>;
default: 3600
context: http, server, location
TTL for cached verification results. After expiry, the next request from the same IP triggers a fresh DNS verification.
bot_verifier_redis_database
syntax: bot_verifier_redis_database <number>;
default: 0
context: http, server, location
Redis database number to use for storing verification results.
bot_verifier_redis_password
syntax: bot_verifier_redis_password <password>;
default: empty
context: http, server, location
Password for Redis authentication. Leave empty if Redis does not require authentication.
Asynchronous DNS Resolution
When the NGINX resolver directive is configured, the module performs DNS lookups asynchronously using NGINX's built-in resolver. This is the recommended configuration for production:
- Non-blocking - DNS lookups do not block NGINX worker processes
- Scalable - handles high traffic without DNS-induced latency spikes
- Graceful timeouts - slow DNS responses do not affect other requests
The verification flow:
- Reverse DNS lookup (PTR record) for the client IP
- Verify the resolved hostname ends with a known provider domain
- Forward DNS lookup (A record) to confirm the IP matches
- Cache the result in Redis
GitHub
You may find additional configuration tips and documentation for this module in the GitHub repository for nginx-module-bot-verifier.