Creating a robots.txt file

Robots.txt - is a file that indicates to the search engine which pages of the site can be indexed, and which cannot. Properly configured robots.txt file enables site indexing and at the same time blocks from indexing those pages of the site that should not be in the search engine index (service pages, search pages, sorting pages, and others).