«Атака ботов»: как ИИ-краулеры ддосят онлайн-проекты

[koreychenko]

А как у ИИ ботов обстоят дела с zip-бомбами? :-)

[ulisma]

nginx conf надо еще прописать, чтобы отдавал 403. И регулярку на названия ботов.

Хотя бы так.

[A_L_I_E_N]

map $http_user_agent $bad_bot {
        default 0;
        ~*(AltaVista|Slurp|BlackWidow|ChinaClaw|Custo|DISCo|Download|Demon|eCatch|EirGrabber|EmailSiphon|EmailWolf|SuperHTTP|Surfbot|WebWhacker|ua_bot|Amazonbot) 1;
        ~*(Express|WebPictures|ExtractorPro|EyeNetIE|FlashGet|GetRight|GetWeb!|GrabNet|Grafula|Go!Zilla|Go-Ahead-Got-It|DuckDuckGo-Favicons-Bot|DuckDuckGo-Bot) 1;
        ~*(HMView|HTTrack|Stripper|Sucker|Indy|InterGET|Ninja|JetCar|Spider|larbin|LeechFTP|Downloader|Navroad|NearSite|NetAnts|tAkeOut|WWWOFFLE|my-tiny-bot|thesis-research-bot) 1;
        ~*(NetSpider|Vampire|NetZIP|Octopus|Offline|PageGrabber|Foto|pavuk|pcBrowser|RealDownload|ReGet|SiteSnagger|SmartDownload|SuperBot|WebSpider|GeedoBot) 1;
        ~*(Teleport|VoidEYE|Collector|WebAuto|WebCopier|WebFetch|WebGo|WebLeacher|WebReaper|WebSauger|eXtractor|Quester|WebStripper|WebZIP|Wget|Widow|Zeus|GPTBot) 1;
        ~*(Twengabot|htmlparser|libwww|Python|urllib|Curl|PycURL|Pyth|WebCollector|WebCopy|webcraw|AspiegelBot|MegaIndex|Copier|netvampire|DISCo|Applebot|AhrefsBot) 1;
        ~*(bingbot|DotBot|Barkrowler|YandexMetrika|YandexMarket|YandexVideo|YandexBot|YandexImage|YandexCalendar|YandexMobileBot|YandexAdNet|YandexAccessibilityBot) 1;
        ~*(YandexBlogs|YandexDirect|Ahrefs|SputnikBot|spbot|DigExt|Sogou|MJ12|majestic12|80legs|SISTRIX|Semrush|Crowsnest|CCBot|TalkTalk|PaperLiBot|peerindex|ia_archiver) 1;
        ~*(Slurp|Aport|NING|JS-Kit|rogerbot|BLEXBot|MJ12bot|Twiceler|Baiduspider|CommentReader|Yeti|discobot|BTWebClient|Tagoobot|Ezooms|igdeSpyder|AhrefsBot) 1;
    }

if ($bad_bot) { return 403; }