###修改添加配置 ###参考：https://www.centos.bz/2018/01/nginx%E6%94%AF%E6%8C%81https%E5%B9%B6%E4%B8%94%E6%94%AF%E6%8C%81%E5%8F%8D%E7%88%AC%E8%99%AB/ vi /etc/nginx/anti_spider.conf ```` #禁止Scrapy等工具的抓取 if ($http_user_agent ~* (Python|Java|Wget|Scrapy|Curl|HttpClient|Spider)) { return 403; } #禁止指定UA及UA为空的访问 if ($http_user_agent ~* "qihoobot|Baiduspider|Googlebot|Googlebot-Mobile|Googlebot-Image|Mediapartners-Google|Adsbot-Google|Feedfetcher-Google|Yahoo! Slurp|Yahoo! Slurp China|YoudaoBot|Sosospider|Sogou spider|Sogou web spider|MSNBot|ia_archiver|Tomato Bot|Catall Spider|AcoiRobot") { return 403; } if ($http_user_agent ~ "WinHttp|WebZIP|FetchURL|node-superagent|java/|FeedDemon|Jullo|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|Java|Feedly|Apache-HttpAsyncClient|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms|BOT/0.1|YandexBot|FlightDeckReports|Linguee Bot|iaskspider^$") { return 403; } #禁止非GET|HEAD|POST方式的抓取 if ($request_method !~ ^(GET|HEAD|POST)$) { return 403; } #屏蔽单个IP的命令是 #deny 123.45.6.7 #封整个段即从123.0.0.1到123.255.255.254的命令 #deny 123.0.0.0/8 #封IP段即从123.45.0.1到123.45.255.254的命令 #deny 124.45.0.0/16 #封IP段即从123.45.6.1到123.45.6.254的命令是 #deny 123.45.6.0/24 # 以下IP皆为流氓 #deny 58.95.66.0/24; ```` ### 添加引导 ```` server{ listen 80; # 本地测试时可以将域名改为: 127.0.0.1 server_name www.lenggirl.com; charset utf-8; access_log /root/logs/nginx/www.lenggirl.com.log; #error_log /data/logs/nginx/www.lenggirl.com.err; location / { proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_redirect off; proxy_pass http://localhost:4000; # 这个就是反爬虫文件了 include /etc/nginx/anti_spider.conf; # } ```` ### 限制浏览器访问网站更目录下增加Robots.txt，放在站点根目录下 if ($http_user_agent ~* "Firefox|MSIE") { return 403; }