txt file, on the other hand, tells engines like google what material to not index (one example is, unique policy internet pages you don’t want to seem in search.) make both of those to speed up the crawling and https://medium.com/@linkdaddy