Useful tools, without the clutter.

Online Tools for Work, Web and Everyday Tasks

Robots.txt Generator

Build a robots.txt file with crawl rules for selected user agents and paths.

Build a robots.txt file with crawl rules for selected user agents and paths. Use it to draft crawler rules for public websites, staging paths or sections you do not want compliant crawlers to request. Remember that robots.txt is not an access-control mechanism and does not make private content secure.

Contact

Missing something?

Feel free to request missing tools or give some feedback using our contact form.

Contact Us