How can you block backlink checking sites, like Majestic, Ahrefs and SEMRush, from crawling or indexing your website?

 Here you go:

Most important bots to block if you like to do:

Majestic SEO -> User-agent: MJ12bot
MOZ OpenSiteExplorer -> User-agent: rogerbot
Ahrefs -> User-agent: Ahrefs

Robots.txt:

  1. User-agent: Rogerbot  
  2. User-agent: Exabot  
  3. User-agent: MJ12bot  
  4. User-agent: Dotbot  
  5. User-agent: Gigabot  
  6. User-agent: AhrefsBot  
  7. User-agent: BlackWidow 
  8. User-agent: ChinaClaw  
  9. User-agent: Custo  
  10. User-agent: DISCo  
  11. User-agent: Download\ Demon  
  12. User-agent: eCatch  
  13. User-agent: EirGrabber  
  14. User-agent: EmailSiphon  
  15. User-agent: EmailWolf  
  16. User-agent: Express\ WebPictures  
  17. User-agent: ExtractorPro  
  18. User-agent: EyeNetIE  
  19. User-agent: FlashGet  
  20. User-agent: GetRight  
  21. User-agent: GetWeb!  
  22. User-agent: Go!Zilla  
  23. User-agent: Go-Ahead-Got-It  
  24. User-agent: GrabNet  
  25. User-agent: Grafula  
  26. User-agent: HMView  
  27. User-agent: HTTrack  
  28. User-agent: Image\ Stripper  
  29. User-agent: Image\ Sucker  
  30. User-agent: Indy\ Library 
  31. User-agent: InterGET  
  32. User-agent: Internet\ Ninja  
  33. User-agent: JetCar  
  34. User-agent: JOC\ Web\ Spider  
  35. User-agent: larbin  
  36. User-agent: LeechFTP  
  37. User-agent: Mass\ Downloader  
  38. User-agent: MIDown\ tool  
  39. User-agent: Mister\ PiX  
  40. User-agent: Navroad  
  41. User-agent: NearSite  
  42. User-agent: NetAnts  
  43. User-agent: NetSpider  
  44. User-agent: Net\ Vampire  
  45. User-agent: NetZIP  
  46. User-agent: Octopus  
  47. User-agent: Offline\ Explorer  
  48. User-agent: Offline\ Navigator  
  49. User-agent: PageGrabber  
  50. User-agent: Papa\ Foto  
  51. User-agent: pavuk  
  52. User-agent: pcBrowser  
  53. User-agent: RealDownload  
  54. User-agent: ReGet  
  55. User-agent: SiteSnagger  
  56. User-agent: SmartDownload  
  57. User-agent: SuperBot  
  58. User-agent: SuperHTTP  
  59. User-agent: Surfbot  
  60. User-agent: tAkeOut  
  61. User-agent: Teleport\ Pro  
  62. User-agent: VoidEYE  
  63. User-agent: Web\ Image\ Collector  
  64. User-agent: Web\ Sucker  
  65. User-agent: WebAuto  
  66. User-agent: WebCopier  
  67. User-agent: WebFetch  
  68. User-agent: WebGo\ IS  
  69. User-agent: WebLeacher  
  70. User-agent: WebReaper  
  71. User-agent: WebSauger  
  72. User-agent: Website\ eXtractor  
  73. User-agent: Website\ Quester  
  74. User-agent: WebStripper  
  75. User-agent: WebWhacker  
  76. User-agent: WebZIP  
  77. User-agent: Wget  
  78. User-agent: Widow  
  79. User-agent: WWWOFFLE  
  80. User-agent: Xaldon\ WebSpider  
  81. User-agent: Zeus 
  82. Disallow: / 

.htaccess:

  1. SetEnvIfNoCase User-Agent .*rogerbot.* bad_bot 
  2. SetEnvIfNoCase User-Agent .*exabot.* bad_bot 
  3. SetEnvIfNoCase User-Agent .*mj12bot.* bad_bot 
  4. SetEnvIfNoCase User-Agent .*dotbot.* bad_bot 
  5. SetEnvIfNoCase User-Agent .*gigabot.* bad_bot 
  6. SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot 
  7. SetEnvIfNoCase User-Agent .*sitebot.* bad_bot 
  8. <Limit GET POST HEAD> 
  9. Order Allow,Deny 
  10. Allow from all 
  11. Deny from env=bad_bot 
  12. </Limit> 

Comments

Post a Comment

Popular posts from this blog

cpanel exam CPSP Answers

 Which of the following options best describes the role of Shared Hosting on a web server? Providing hosting for a single website owned by different people on multiple servers. Providing hosting for a single website owned by one person on one server. Providing hosting for multiple websites owned by different people on one server. Providing hosting for multiple websites owned by one person on one server. 3 Which of the following capabilities can a cPanel account user perform easily from within the cPanel account interface without the aid of a system administrator? File and configuration management. Relocating the physical server. Upgrading server hardware and equipment. Installing new database software. 1 File and configuration management. Which of the following options best describes a core benefit to using cPanel & WHM, as a web hosting provider operating on a cPanel & WHM environment? cPanel users are able to create new virtualization resources to help host their web appl...
Read more

How to install zimbra collaboration suite 8.8.11 on CentOS 7

Check system requirements  https://wiki.zimbra.com/wiki/Zimbra_Releases/8.7.0/Single_Server_Installation#System_Requirements prerequisites Before we get started, here are a few things that are required before we proceed with the installation. A clean installation of Centos 7 A Fully Qualified Domain (FQDN) for your server(Example mail.geekybum.com) An external DNS server with both A and MX records for your server to point to your Zimbra mail server IP Address A static IP Address assigned to network interface. Step 1: Root login to server through SSH Step 2: Install required packages for zimbra coloboration suite installation yum -y install unzip net-tools sysstat openssh-clients perl-core libaio nmap-ncat libstdc++.so Initial server setup To get started we need to configure a hostname for our server and a static IP address. vi /etc/hosts 135.125.30.56 mail.geekybum.com mail Replace the system hostname and FQDN values accordingly in order to match your ow...
Read more

awstats installation

Set Up AWStats for Nginx on Ubuntu sudo apt-get install awstats add access log file to generate stats using this server { # ... all your other config ... access_log /var/log/nginx/yourdomain.com.access.log ; # ... all your other config ... } /etc/nginx/sites-available vi awstats.parleproducts.com.conf cd /etc/awstats/ vi awstats.parleproducts.com.conf #path to your nginx log file LogFile="/var/log/nginx/parleproducts.com.access.log" # Domain of your vhost SiteDomain="parleproducts.com" # Directory where to store the awstats data DirData="/var/lib/awstats/" # Other domains/subdomain you want included from your logs, for example the www subdomain HostAliases="www.parleproducts.com" # If you customized your log format above add this line: LogFormat = ""%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot"" # If you did not, uncomment and use this line: #...
Read more