Generate Robots.txt
Download Robots.txt
Generate Robots.txt
Default rule – All the robots will be
Allowed
Refused
Crawl delay
No delay
5 seconds
10 seconds
20 seconds
60 seconds
120 seconds
Site maps
Restricted urls/directories (up to 6)
Restricted url 0
Restricted url 1
Restricted url 2
Restricted url 3
Restricted url 4
Restricted url 5
Allowed Robots*
Google
Same as default
Allowed
Refused
Google Image
Same as default
Allowed
Refused
Google Mobile
Same as default
Allowed
Refused
MSN Search
Same as default
Allowed
Refused
Yahoo
Same as default
Allowed
Refused
Yahoo MM
Same as default
Allowed
Refused
Yahoo Blogs
Same as default
Allowed
Refused
Ask/Teoma
Same as default
Allowed
Refused
GigaBlast
Same as default
Allowed
Refused
DMZO Checker
Same as default
Allowed
Refused
Nutch
Same as default
Allowed
Refused
Alexa/Wayback
Same as default
Allowed
Refused
Baidu
Same as default
Allowed
Refused
Naver
Same as default
Allowed
Refused
MSN PicSearch
Same as default
Allowed
Refused
Disallowed Robots*
asterias
Same as default
Allowed
Refused
BackDoorBot/1.0
Same as default
Allowed
Refused
Black Hole
Same as default
Allowed
Refused
BlowFish/1.0
Same as default
Allowed
Refused
BotALot
Same as default
Allowed
Refused
BuiltBotTough
Same as default
Allowed
Refused
Bullseye/1.0
Same as default
Allowed
Refused
BunnySlippers
Same as default
Allowed
Refused
Cegbfeieh
Same as default
Allowed
Refused
CheeseBot
Same as default
Allowed
Refused
CherryPicker
Same as default
Allowed
Refused
CherryPickerElite/1.0
Same as default
Allowed
Refused
CherryPickerSE/1.0
Same as default
Allowed
Refused
CopyRightCheck
Same as default
Allowed
Refused
cosmos
Same as default
Allowed
Refused
Crescent
Same as default
Allowed
Refused
Crescent Internet ToolPak HTTP OLE Control v.1.0
Same as default
Allowed
Refused
DittoSpyder
Same as default
Allowed
Refused
EmailCollector
Same as default
Allowed
Refused
EmailSiphon
Same as default
Allowed
Refused
EmailWolf
Same as default
Allowed
Refused
EroCrawler
Same as default
Allowed
Refused
ExtractorPro
Same as default
Allowed
Refused
Foobot
Same as default
Allowed
Refused
Harvest/1.5
Same as default
Allowed
Refused
hloader
Same as default
Allowed
Refused
httplib
Same as default
Allowed
Refused
humanlinks
Same as default
Allowed
Refused
ia_archiver
Same as default
Allowed
Refused
InfoNaviRobot
Same as default
Allowed
Refused
JennyBot
Same as default
Allowed
Refused
Kenjin Spider
Same as default
Allowed
Refused
Keyword Density/0.9
Same as default
Allowed
Refused
LexiBot
Same as default
Allowed
Refused
libWeb/clsHTTP
Same as default
Allowed
Refused
LinkextractorPro
Same as default
Allowed
Refused
LinkScan/8.1a Unix
Same as default
Allowed
Refused
LinkWalker
Same as default
Allowed
Refused
LNSpiderguy
Same as default
Allowed
Refused
lwp-trivial
Same as default
Allowed
Refused
lwp-trivial/1.34
Same as default
Allowed
Refused
Mata Hari
Same as default
Allowed
Refused
Microsoft URL Control - 5.01.4511
Same as default
Allowed
Refused
Microsoft URL Control - 6.00.8169
Same as default
Allowed
Refused
MIIxpc
Same as default
Allowed
Refused
MIIxpc/4.2
Same as default
Allowed
Refused
Mister PiX
Same as default
Allowed
Refused
moget
Same as default
Allowed
Refused
moget/2.1
Same as default
Allowed
Refused
mozilla/4
Same as default
Allowed
Refused
Mozilla/4.0 (compatible; BullsEye; Windows 95)
Same as default
Allowed
Refused
Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
Same as default
Allowed
Refused
Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
Same as default
Allowed
Refused
Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
Same as default
Allowed
Refused
Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
Same as default
Allowed
Refused
Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
Same as default
Allowed
Refused
Mozilla/4.0 (compatible; MSIE 4.0; Windows ME)
Same as default
Allowed
Refused
mozilla/5
Same as default
Allowed
Refused
NetAnts
Same as default
Allowed
Refused
NICErsPRO
Same as default
Allowed
Refused
Offline Explorer
Same as default
Allowed
Refused
Openfind
Same as default
Allowed
Refused
Openfind data gathere
Same as default
Allowed
Refused
ProPowerBot/2.14
Same as default
Allowed
Refused
ProWebWalker
Same as default
Allowed
Refused
QueryN Metasearch
Same as default
Allowed
Refused
RepoMonkey
Same as default
Allowed
Refused
RepoMonkey Bait & Tackle/v1.01
Same as default
Allowed
Refused
RMA
Same as default
Allowed
Refused
SiteSnagger
Same as default
Allowed
Refused
SpankBot
Same as default
Allowed
Refused
spanner
Same as default
Allowed
Refused
suzuran
Same as default
Allowed
Refused
Szukacz/1.4
Same as default
Allowed
Refused
Teleport
Same as default
Allowed
Refused
TeleportPro
Same as default
Allowed
Refused
Telesoft
Same as default
Allowed
Refused
The Intraformant
Same as default
Allowed
Refused
TheNomad
Same as default
Allowed
Refused
TightTwatBot
Same as default
Allowed
Refused
Titan
Same as default
Allowed
Refused
toCrawl/UrlDispatcher
Same as default
Allowed
Refused
True_Robot
Same as default
Allowed
Refused
True_Robot/1.0
Same as default
Allowed
Refused
turingos
Same as default
Allowed
Refused
URLy Warning
Same as default
Allowed
Refused
VCI
Same as default
Allowed
Refused
VCI WebViewer VCI WebViewer Win32
Same as default
Allowed
Refused
Web Image Collector
Same as default
Allowed
Refused
WebAuto
Same as default
Allowed
Refused
WebBandit
Same as default
Allowed
Refused
WebBandit/3.50
Same as default
Allowed
Refused
WebCopier
Same as default
Allowed
Refused
WebEnhancer
Same as default
Allowed
Refused
WebmasterWorldForumBot
Same as default
Allowed
Refused
WebSauger
Same as default
Allowed
Refused
Website Quester
Same as default
Allowed
Refused
Webster Pro
Same as default
Allowed
Refused
WebStripper
Same as default
Allowed
Refused
WebZip
Same as default
Allowed
Refused
WebZip/4.0
Same as default
Allowed
Refused
Wget
Same as default
Allowed
Refused
Wget/1.5.3
Same as default
Allowed
Refused
Wget/1.6
Same as default
Allowed
Refused
WWW-Collector-E
Same as default
Allowed
Refused
Xenu's
Same as default
Allowed
Refused
Xenu's Link Sleuth 1.1c
Same as default
Allowed
Refused
Zeus
Same as default
Allowed
Refused
Zeus 32297 Webster Pro V2.9 Win32
Same as default
Allowed
Refused
rogerbot
Same as default
Allowed
Refused
mj12bot
Same as default
Allowed
Refused
dotbot
Same as default
Allowed
Refused
ahrefsbot
Same as default
Allowed
Refused
Robots.txt Preview
# my robots.txt