It’s important to have a robots.txt file. This file tells the search engine spider which pages to crawl.

Click here to generate robots.txt file for your website.