How can you block backlink checking sites, like Majestic, Ahrefs and SEMRush, from crawling or indexing your website?

 Here you go:

Most important bots to block if you like to do:

Majestic SEO -> User-agent: MJ12bot
MOZ OpenSiteExplorer -> 
User-agent: rogerbot
Ahrefs -> 
User-agent: Ahrefs

Robots.txt:

  1. User-agent: Rogerbot  
  2. User-agent: Exabot  
  3. User-agent: MJ12bot  
  4. User-agent: Dotbot  
  5. User-agent: Gigabot  
  6. User-agent: AhrefsBot  
  7. User-agent: BlackWidow 
  8. User-agent: ChinaClaw  
  9. User-agent: Custo  
  10. User-agent: DISCo  
  11. User-agent: Download\ Demon  
  12. User-agent: eCatch  
  13. User-agent: EirGrabber  
  14. User-agent: EmailSiphon  
  15. User-agent: EmailWolf  
  16. User-agent: Express\ WebPictures  
  17. User-agent: ExtractorPro  
  18. User-agent: EyeNetIE  
  19. User-agent: FlashGet  
  20. User-agent: GetRight  
  21. User-agent: GetWeb!  
  22. User-agent: Go!Zilla  
  23. User-agent: Go-Ahead-Got-It  
  24. User-agent: GrabNet  
  25. User-agent: Grafula  
  26. User-agent: HMView  
  27. User-agent: HTTrack  
  28. User-agent: Image\ Stripper  
  29. User-agent: Image\ Sucker  
  30. User-agent: Indy\ Library 
  31. User-agent: InterGET  
  32. User-agent: Internet\ Ninja  
  33. User-agent: JetCar  
  34. User-agent: JOC\ Web\ Spider  
  35. User-agent: larbin  
  36. User-agent: LeechFTP  
  37. User-agent: Mass\ Downloader  
  38. User-agent: MIDown\ tool  
  39. User-agent: Mister\ PiX  
  40. User-agent: Navroad  
  41. User-agent: NearSite  
  42. User-agent: NetAnts  
  43. User-agent: NetSpider  
  44. User-agent: Net\ Vampire  
  45. User-agent: NetZIP  
  46. User-agent: Octopus  
  47. User-agent: Offline\ Explorer  
  48. User-agent: Offline\ Navigator  
  49. User-agent: PageGrabber  
  50. User-agent: Papa\ Foto  
  51. User-agent: pavuk  
  52. User-agent: pcBrowser  
  53. User-agent: RealDownload  
  54. User-agent: ReGet  
  55. User-agent: SiteSnagger  
  56. User-agent: SmartDownload  
  57. User-agent: SuperBot  
  58. User-agent: SuperHTTP  
  59. User-agent: Surfbot  
  60. User-agent: tAkeOut  
  61. User-agent: Teleport\ Pro  
  62. User-agent: VoidEYE  
  63. User-agent: Web\ Image\ Collector  
  64. User-agent: Web\ Sucker  
  65. User-agent: WebAuto  
  66. User-agent: WebCopier  
  67. User-agent: WebFetch  
  68. User-agent: WebGo\ IS  
  69. User-agent: WebLeacher  
  70. User-agent: WebReaper  
  71. User-agent: WebSauger  
  72. User-agent: Website\ eXtractor  
  73. User-agent: Website\ Quester  
  74. User-agent: WebStripper  
  75. User-agent: WebWhacker  
  76. User-agent: WebZIP  
  77. User-agent: Wget  
  78. User-agent: Widow  
  79. User-agent: WWWOFFLE  
  80. User-agent: Xaldon\ WebSpider  
  81. User-agent: Zeus 
  82. Disallow: / 

.htaccess:

  1. SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot 
  2. SetEnvIfNoCase User-Agent .*exabot.* bad_bot 
  3. SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot 
  4. SetEnvIfNoCase User-Agent .*dotbot.* bad_bot 
  5. SetEnvIfNoCase User-Agent .*gigabot.* bad_bot 
  6. SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot 
  7. SetEnvIfNoCase User-Agent .*sitebot.* bad_bot 
  8. <Limit GET POST HEAD> 
  9. Order Allow,Deny 
  10. Allow from all 
  11. Deny from env=bad_bot 
  12. </Limit> 

Comments

Popular posts from this blog

cpanel exam CPSP Answers

How to install zimbra collaboration suite 8.8.11 on CentOS 7

awstats installation