_ TWF  _ Code Chap 23

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
HOME | BACK | NEXT | MENU | SEARCH | ORDER | COMMENT | HELPINFO -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

23code01.pl


#robots.txt for http://www.outer.net/

User-agent:*

Disallow:/cypherdocs/map/   #this is a private URL space

Disallow:/tmp/              #temporary files, please ignore

# Interpretation:

# This file applies to all User-agents ('*')--that is, all robots

# All URLs within the /cypherdocs/map/ hierarchy are disallowed

# All URLs within the /tmp/hierarchy are disallowed



#robots.txt for http://www.io.com/

User-agent: *

Disallow: /docs/            #system administrator URL space



User-agent: localbot

Disallow:

# Interpretation

# This file applies to all external User-agents ('*')

# Notice the blank line that separates the first & second records

# The second, empty "Disallow" grants the User-agent "localbot"

# access to all URLs




-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
HOME | BACK | NEXT | MENU | SEARCH | ORDER | COMMENT | HELPINFO -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

E-Mail: The Web Foundations at twf@lanw.com
URL: http://www.lanw.com/twf/codech23.htm
Text - Copyright © 1995, Ed Tittel, Mark Gaither, Sebastian Hassinger, & Mike Erwin.
Web Layout - Copyright © 1995, LANWrights & IMPACT Online.
Revised -- February 15th, 1996 [James Michael Stewart - WebMaster - IMPACT Online]

HTML 2.0 Checked!