block crawling of a page with robots.txt


create a robots.txt block thes pages, https://web-stories.in/c and https://web-stories.in/shop.php





User-agent: *
Disallow: /c
Disallow: /shop.php





This robots.txt file will block all search engines from crawling the pages /c and /shop.php. As a result, these pages will not be indexed by search engines and will not appear in search results.
To create a robots.txt file, you can use a text editor to create a new file named robots.txt and add the above code to it. Then, you can upload the file to the root directory of your website.
Once the robots.txt file is in place, search engines will start crawling your website and will respect the instructions in the file. This will prevent them from crawling the pages /c and /shop.php, and these pages will not be indexed by search engines.
























Comments

Post a Comment

Popular posts from this blog

cpanel exam CPSP Answers

 Which of the following options best describes the role of Shared Hosting on a web server? Providing hosting for a single website owned by different people on multiple servers. Providing hosting for a single website owned by one person on one server. Providing hosting for multiple websites owned by different people on one server. Providing hosting for multiple websites owned by one person on one server. 3 Which of the following capabilities can a cPanel account user perform easily from within the cPanel account interface without the aid of a system administrator? File and configuration management. Relocating the physical server. Upgrading server hardware and equipment. Installing new database software. 1 File and configuration management. Which of the following options best describes a core benefit to using cPanel & WHM, as a web hosting provider operating on a cPanel & WHM environment? cPanel users are able to create new virtualization resources to help host their web applicat
Read more

How to install zimbra collaboration suite 8.8.11 on CentOS 7

Check system requirements  https://wiki.zimbra.com/wiki/Zimbra_Releases/8.7.0/Single_Server_Installation#System_Requirements prerequisites Before we get started, here are a few things that are required before we proceed with the installation. A clean installation of Centos 7 A Fully Qualified Domain (FQDN) for your server(Example mail.geekybum.com) An external DNS server with both A and MX records for your server to point to your Zimbra mail server IP Address A static IP Address assigned to network interface. Step 1: Root login to server through SSH Step 2: Install required packages for zimbra coloboration suite installation yum -y install unzip net-tools sysstat openssh-clients perl-core libaio nmap-ncat libstdc++.so Initial server setup To get started we need to configure a hostname for our server and a static IP address. vi /etc/hosts 135.125.30.56 mail.geekybum.com mail Replace the system hostname and FQDN values accordingly in order to match your own do
Read more

awstats installation

Set Up AWStats for Nginx on Ubuntu sudo apt-get install awstats add access log file to generate stats using this server { # ... all your other config ... access_log /var/log/nginx/yourdomain.com.access.log ; # ... all your other config ... } /etc/nginx/sites-available vi awstats.parleproducts.com.conf cd /etc/awstats/ vi awstats.parleproducts.com.conf #path to your nginx log file LogFile="/var/log/nginx/parleproducts.com.access.log" # Domain of your vhost SiteDomain="parleproducts.com" # Directory where to store the awstats data DirData="/var/lib/awstats/" # Other domains/subdomain you want included from your logs, for example the www subdomain HostAliases="www.parleproducts.com" # If you customized your log format above add this line: LogFormat = ""%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot"" # If you did not, uncomment and use this line: #
Read more