{"id":1232,"date":"2008-10-19T08:13:40","date_gmt":"2008-10-19T15:13:40","guid":{"rendered":"http:\/\/eaf.net\/mvp\/?p=1232"},"modified":"2011-11-07T07:25:34","modified_gmt":"2011-11-07T15:25:34","slug":"yahoo-slurp-and-me","status":"publish","type":"post","link":"https:\/\/www.eaf.net\/mvp\/2008\/yahoo-slurp-and-me\/","title":{"rendered":"Yahoo! Slurp and Me"},"content":{"rendered":"

They helped cause a spike on this site that led to my host suspending the domain.<\/p>\n

After at least 30 hours of being suspended, I was allowed to bring the domain back online. Since then, I’ve been monitoring its bandwidth consumption. And particularly watching two bots: Googlebot<\/i> and Yahoo! Slurp<\/i>.<\/p>\n

I also set up robots.txt to severely restrict bot access to this domain.<\/p>\n

Googlebot is behaving; Slurp appears not to be.<\/p>\n


Robots\/Spiders at 6:53 am on 10\/18\/08
\n  Yahoo Slurp  7785+312  156.18 MB  09:46
\n  Googlebot    4411+33   111.47 MB  02:42<\/code><\/p>\n

Robots\/Spiders at 7:20 am on 10\/19\/08
\n  Yahoo Slurp  8357+334  164.00 MB  10:05
\n  Googlebot    4413+35   111.53 MB  03:59<\/code><\/p><\/blockquote>\n

I checked my recent accesses…and found a whole bunch by Yahoo. \ud83d\ude41<\/p>\n

<\/p>\n

At this point, I’m trying to give them the benefit of the doubt because my robots.txt file had a long disallow section for all bots and another special very short one with no disallows for Slurp because I figured they would honor both. Apparently they don’t. So I modified my robots file to remove the special Slurp section.<\/p>\n

Now we’ll see what happens.<\/p>\n

If I knew what IPs to ban in htaccess, I would do that as well!<\/p>\n

Oh…and I’m posting about this here and on this day for the record<\/b>. Now I can point my virtual server provider to this blog post. The surge in traffic ain’t my fault, guys!<\/p>\n

\":mrgreen:\"<\/p>\n

October 20 update:<\/b><\/p>\n

OK, here’s what I’m banning so far in my .htaccess file:<\/p>\n

RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Bot\\ mailto:craftbot@yahoo.com [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Custo [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Download\\ Demon [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Express\\ WebPictures [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
\nRewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Image\\ Stripper [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Image\\ Sucker [OR]
\nRewriteCond %{HTTP_USER_AGENT} Indy\\ Library [NC,OR]
\nRewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Internet\\ Ninja [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^JOC\\ Web\\ Spider [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^larbin [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Mass\\ Downloader [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^MIDown\\ tool [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Mister\\ PiX [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Net\\ Vampire [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Offline\\ Explorer [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Offline\\ Navigator [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Papa\\ Foto [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^pavuk [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Teleport\\ Pro [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Web\\ Image\\ Collector [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Web\\ Sucker [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^WebGo\\ IS [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Website\\ eXtractor [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Website\\ Quester [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Xaldon\\ WebSpider [OR]
\nRewriteCond %{HTTP_USER_AGENT} ^Zeus<\/p>\n

deny from 8.11.2.98
\ndeny from 41.202.78.138
\ndeny from 62.12.137.20
\ndeny from 62.37.113.210
\ndeny from 62.37.112.104
\ndeny from 62.37.117.18
\ndeny from 62.4.64.119
\ndeny from 64.34.161.151
\ndeny from 66.198.41.11
\ndeny from 66.7.210.15
\ndeny from 67.227.134.4
\ndeny from 69.42.90.166
\ndeny from 72.37.152.51
\ndeny from 74.208.14.215
\ndeny from 76.79.154.182
\ndeny from 77.88.26.26
\ndeny from 77.91.224.
\ndeny from 78.86.153.121
\ndeny from 83.133.114.2
\ndeny from 83.133.125.166
\ndeny from 85.124.67.254
\ndeny from 85.92.70.252
\ndeny from 85.92.71.147
\ndeny from 87.224.173.
\ndeny from 89.208.32.109
\ndeny from 91.205.124.
\ndeny from 189.30.142.213
\ndeny from 196.35.158.180
\ndeny from 200.63.42.136
\ndeny from 201.66.127.38
\ndeny from 202.75.62.162
\ndeny from 205.234.140.219
\ndeny from 207.182.151.154
\ndeny from 208.109.181.121
\ndeny from 208.148.196.71
\ndeny from 209.200.18.
\ndeny from 209.200.48.
\ndeny from 209.216.249.184
\ndeny from 212.116.219.170
\ndeny from 212.58.4.129
\ndeny from 213.4.106.85
\ndeny from 213.134.144.134
\ndeny from 216.130.187.100
\ndeny from 220.225.241.4<\/p>\n","protected":false},"excerpt":{"rendered":"

They helped cause a spike on this site that led to my host suspending the domain. After at least 30 hours of being suspended, I was allowed to bring the domain back online. Since then, I’ve been monitoring its bandwidth consumption. And particularly watching two bots: Googlebot and Yahoo! Slurp. I also set up robots.txt … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[13],"tags":[424,716],"class_list":["post-1232","post","type-post","status-publish","format-standard","hentry","category-tech-stuff","tag-security","tag-wordpress"],"jetpack_publicize_connections":[],"aioseo_notices":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/prJUJ-jS","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.eaf.net\/mvp\/wp-json\/wp\/v2\/posts\/1232","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.eaf.net\/mvp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.eaf.net\/mvp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.eaf.net\/mvp\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.eaf.net\/mvp\/wp-json\/wp\/v2\/comments?post=1232"}],"version-history":[{"count":0,"href":"https:\/\/www.eaf.net\/mvp\/wp-json\/wp\/v2\/posts\/1232\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.eaf.net\/mvp\/wp-json\/wp\/v2\/media?parent=1232"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.eaf.net\/mvp\/wp-json\/wp\/v2\/categories?post=1232"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.eaf.net\/mvp\/wp-json\/wp\/v2\/tags?post=1232"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}