Configuration of the robots.txt file
A robots.txt file is created that manages the indexation of the site based on the current structure of the site and its general state.
Lvov 
A little busy