$is_ai_crawler
The $is_ai_crawler variable is set by the device-type module to indicate whether a request is from an AI training or search crawler. It helps in identifying bots like GPTBot or ClaudeBot.
Type
Boolean (1/0)
Possible values
1: The request is from an AI crawler.0: The request is not from an AI crawler.
Examples
Access Control
Block AI crawlers from accessing certain paths.
location /sensitive-data/ {
if ($is_ai_crawler) {
return 403 "Access denied for AI crawlers";
}
proxy_pass http://backend;
}
Custom Header for Backend
Add a custom header to inform the backend about AI crawler requests.
location /api/ {
proxy_set_header X-Is-AI-Crawler $is_ai_crawler;
proxy_pass http://api_backend;
}
Logging AI Crawler Requests
Log requests from AI crawlers for monitoring purposes.
log_format ai_crawler '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" '
'AI_Crawler:$is_ai_crawler';
access_log /var/log/nginx/ai_crawler.log ai_crawler;