From: jfricker@vertexgroup.com (John F. Fricker)
Date: Tue, 16 Jan 96 15:02:39 PST
To: cypherpunks@toad.com
Subject: Re: Spiderspace
Message-ID: <2.2.32.19960116231023.00616ea8@vertexgroup.com>
MIME-Version: 1.0
Content-Type: text/plain


At 11:38 AM 1/16/96 -0800, you wrote:
>
>I've been thinking a lot about the problems and opportunities that are
>coming up as more and more "spiders" (Web searchers, crawlers) are indexing
>directories and files on systems they can find.

<a snip of agreement>


I haven't checked my logs for Alta Vista but many of the spiders do follow the robot exclusion standard. A simple text file that should be retrieved first by any spider that explains where one may not go.

The standard is fully explained at:

http://info.webcrawler.com/mak/projects/robots/norobots.html

But this does nothing for those items sitting in more traditional public space. Obscurity is no longer security.