Robots.txt Generator

Generate a robots.txt file to control search engine crawling

General Settings

Full URL to your XML sitemap

Search Robots

Set individual access rules per robot. "Default" means the robot follows the general setting above.

Google
Googlebot
Google Image
Googlebot-Image
Google Mobile
Googlebot-Mobile
MSN Search
msnbot
Yahoo
Slurp
Yahoo MM
Yahoo-MMCrawler
Yahoo Blogs
Yahoo-Blogs
Ask/Teoma
Teoma
GigaBlast
Gigabot
DMOZ Checker
Robozilla
Nutch
Nutch
Alexa/Wayback
ia_archiver
Baidu
Baiduspider
Naver
Yeti
MSN PicSearch
psbot
Restricted Directories

The path is relative to root and must contain a trailing slash "/"

Generated robots.txt
© Toolivy. v1.0
Powered with by RL
Code snippet