What is robots.txt
A robots.txt file is a text file that tells web crawlers (like Googlebot) which parts of your website they are allowed to access. It is a simple way to control how your website is indexed by search engines.
How to add/edit your robots.txt file?
From the Responsive website section of the left menu, expand Website and then select Basic info.
Select Asset Details.
Scroll down to the Robots file and format Add your robots.txt file here in .txt format.
What should your robots.txt contain?
The syntax for a robots.txt file is as follows:
The User-agent line specifies the search engine bot that the directive applies to.
The Disallow line specifies a path that the search engine bot should not crawl.
The Allow line specifies a path that the search engine bot is allowed to crawl.
The Sitemap line specifies the URL of a sitemap file that the search engine bot can use to crawl your website