Please specify URL to crawl. (example)
Quick robots.txt checker to test which robots are disallowed or allowed to index/archive contents. recognizes robots.txt and <meta> tag specified by Google.
Bookmarklet: test robots.txt
Parmalink: http://forbiddenrobots.usb0.net/