Created OnApril 5, 2019byadmin You are here: Main SEO Importance of robots.txt ← All Topics It’s important to have a robots.txt file. This file tells the search engine spider which pages to crawl. Click here to generate robots.txt file for your website.