User-agent: * Disallow: *mailto* Disallow: */feed/ Disallow: */admin/ Disallow: */wp-admin/ Disallow: */misc/ Disallow: */myerror/ Disallow: /cgi-bin/ Disallow: /wp-login* Disallow: /category/ Disallow: /2008/ Disallow: /2009/ Disallow: /tag/ # disable following crawlers User-agent: Steeler Disallow: / # reason: disregard this robot rule, so just block it now. #User-agent: MJ12bot #Disallow: / User-agent: IRLbot Disallow: / User-agent: yetibot Disallow: /