It’s important to have a robots.txt file. This file tells the search engine spider which pages to crawl. Click here to generate robots.txt file for your website.
