Technique

The robots.txt file explained

What is a robots.txt file? Search engine through a program robot (also known as spider), automatically access the Internet page and access to web information. You can create a plain...