From: Tim May <tcmay@got.net>
Date: Fri, 30 Jan 1998 02:51:26 +0800
To: cypherpunks@Algebra.COM
Subject: Persistent URLs Considered Useful
In-Reply-To: <9801291747.AA26773@mentat.com>
Message-ID: <v03102800b0f67987a920@[207.167.93.63]>
MIME-Version: 1.0
Content-Type: text/plain



At 9:47 AM -0800 1/29/98, Jim Gillogly wrote:

>BTW, I went to look this up in the Cyphernomicon (I sorta think it's
>reffed in there), but the first 4 sites I saw on Altavista were all
>dead-end broken links.  The Web's ripping... what's the current
>Preferred URL?

I have no idea. Sometimes I find a site has it, sometimes not. (I'm not
interested in keeping my tens of megs of postings on a Web page. Sosumi.)

The issue of "URL decay" is a very serious one, affecting directly the use
of the Web for footnote citations, legal citations, references, etc.
Scientific or academic articles cannot reliably cite URLs, as they are
likely to decay or vanish or become corrupted over a matter of months, let
alone the years or decades that a technical or academic paper is expected
to last and be read. Call it an archivist's nightmare.

A couple of years ago I floated an idea out to a handful of Bay Area
friends and Cypherpunks, about an idea for an "Eternity.com" service which
would act as a kind of "vanity press" for authors, professors, researchers,
etc., who wanted to know that a document and URL would have a long
persistence, possibly an indefinite persistence.

(At the time I used the "Eternity" name I was not consciously aware of Ross
Anderson's work on his "Eternity" system, though I may have inadvertently
been inspired by his name, which I may have heard on the list or elsewhere.
My notion was a bit different from either his Eternity system or any of the
recent variants, but the name was based on the same idea of perpetual
storage. However, unlike my BlackNet idea, this particular "Eternity" was
not focussed on contraband, illegal, controversial, or black market
information being distributed and preserved. In fact, the expected
customers were mundane academics and corporate users...or vanity
users...anyone, basically, who wanted to know that a paper or document of
theirs could be reliably cited by others and that the citations would not
exhibit the "URL decay" so commonly seen today.)

Like I said, I sent this to a handful of Bay Area friends and Cypherpunks,
mainly to see if they had any comments or interest. I chose at that time
not to send my idea out to the Cypherpunks list, as I had some thoughts
about maybe trying to commercialize the idea....

However, I haven't, so I may as well send this idea along.

Here is the piece I sent out in late '95:

Return-Path: tcmay@got.net
Received: from [205.199.118.202] (tcmay.got.net [205.199.118.202]) by
you.got.net (8.6.9/8.6.9) with SMTP id OAA05317 for <tcmay@got.net>; Fri,
29 Dec 1995 14:30:25 -0800
Date: Fri, 29 Dec 1995 14:30:25 -0800
X-Sender: tcmay@mail.got.net
Message-Id: <ad0998a917021004d556@[205.199.118.202]>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
To: tcmay@got.net
From: tcmay@got.net (Timothy C. May)
Subject: Eternity.com announces "Eternal URLs"--version 1.02
X-UIDL: 820276233.000


***Press Release***

Santa Cruz, CA. Eternity, Inc. is pleased to announce the availability of
its permanent Uniform Resource Locater (URL) facility. For a fee, Eternity
will store a file for a year, for several years, or for "eternity."
Eternity will ensure that backups are escrowed in the event of
interruptions at Eternity or even the demise of Eternity as an entity.
Visit us at

http://www.eternity.com/~products/

***End***

Well, not really. But an example makes a point succinctly.

There's a growing need for "eternal URLs," URLs that have enough
persistence to last for a very long time, possibly for several decades.
(And with declining storage costs, "several years" implies "eternity" only
costs 10% more.)

Here are some reasons:

* Today's URLs are transient, fluid. Accounts go away and the URL no longer
works, e.g., the large number of "Can't access" errors when Web surfing.

(Many of these errors are because a server or account has been deliberately
been taken down, for whatever reasons. Often because of overloading, which
is a separable issue from the issue of permanence--overloading can be
handled with access fees, for example.)

* Web citations. Michael Froomkin, and others, have been raising the issue
of using the Web for legal citations. A real problem if the citations don't
persist for very long. (And I started seriously thinking about this Web
transience issue when I wrote a reply to him pointing out the utter
transience of most URLs, the "decay time" of valid URLs and how tough it
would be to rely on the Web for citations which are intended to be valid
for years or even decades.)

* Many links have a "single-point failure" built in. Let me give an example:

- my Cyphernomicon exists in an HTML version (nicely done by J. Rochkind)
at the URL "http://www.oberlin.edu/~brchkind/cyphernomicon/".

- lots of other Web pages have links to this URL. In fact, nearly every
page that mentions the Cyphernomicon points to this URL!

- so, what happens when Jonathan's account--at Oberlin I presume--goes
away? What happens when he gets tired of maintaining it, or graduates, or
whatever?

- answer: all the pages that point to it now come up only with the typical
errors.

- it's unlikely that all or even most of the pages with pointers to this
now-gone URL will update them. I call this "Web decay."

Thus, we see the increasing breakage of links. (I don't know about you
folks, but in my Web surfing I find more and more links broken every
day....)

A market fix might be the following:

- a site operator offers "archival" storage, perhaps/probably for a fee

- files could be placed in this archival storage for some fee for a given
amount of time. (Given the declining cost of storage, a user might be able
to economically buy "permanent" storage, sort of a "discounted future
value" approach. Thus, I might pay $20 to store the Cyphernomicon for a
year, $30 for 3 years, and $50 for "eternity."

- he may also charge digital cash for access (a separable issue, but worth
mentioning)

- example URL: "http://www.eternity.com/~cypherpunks/cyphernomicon/"

- the site, "eternity.com" in this example, would ensure that the document
remains mounted and available for years, decades, etc. This could be done
by using backup machines, copies of optical disks at services which agree
(for a fee) to keep backups and mount them in the event of
disruption/bankruptcy/etc. of "eternity.com," and so on. (Obviously a fee
structure could include issues of file size, latency of access, policies
for public hits on the site, etc.)

- services like this (and I expect more to appear) may have cross-backup
provisions, or arrangements to take over the good name and good will, and
of course the files, of services which vanish or go bankrupt. (All sorts of
messy details, but familiar to lawyers handling escrow matters.)

- there are obvious similarities with "archival storage" services (the
"data vaults" and salt mine companies). In this case, the archival storage
also has access via the Web attached. (And yes, an obvious wrinkle is to
sell archival storage per se, with access a separate issue: access could be
via passwords/crypto, or via paid access, URLs, etc. All separable. But the
"market focus" on "eternal URLs" is a powerful focus, likely to quickly
generate a fair amount of business.)

(Strategies for site-mirroring of URLs, where the same URL actually
involves multiple sites, is another approach. I'm not following that area
too closely, so I won't discuss it here. It may be a workable alternative.)

Another alternative is for services to arise which act as redirectors of
Web accesses. Without actually storing the Cyphernomicon file, for example,
they tell users where it actually is. (Not that much different from
Infoseek, DejaNews, Alta Vista, etc., except that those services merely
index existing pages, which may have huge numbers of corrupt links, while
the service I am proposing would take active steps to ensure valid copies
can be accessed...probably too much work to be economical, which is why I
prefer the "Eternity.com" model: the owner of a URL, or an interested party
who wishes to pay to store a copy, takes active steps to ensure a permanent
copy exists.

There are all sorts of wrinkles of this idea, such as:

-- newer versions of a file, e.g., the "Cyphernomicon v.1.5," are either
stored separately, with pointers added to the first file, or are appended.
I favor the "pure archive" approach of always having the earlier versions
stored. ("Once stored, it is never forgotten.") For example:

"http://www.eternity.com/~cypherpunks/cyphernomicon/0.666" (the original)
"http://www.eternity.com/~cypherpunks/cyphernomicon/1.5"
"http://www.eternity.com/~cypherpunks/cyphernomicon/2.0"
...etc....

-- obviously files could be stored in various ways, depending on fees per
storage and fees per access. Older versions might be archived on DAT (or
its 2005 equivalent), more recent versions might be on DVDs, CD-ROMs, etc.,
and heavily-accessed files might be on magnetic disk. All a matter of
pricing, usage, market issues.

-- jukeboxes of CD-ROMs, DATs, and DVDs should make storage of "archived
Web sites" very cheap.

-- such a service could also be a protection against political pressure:
once a file is stored, perhaps in multiple national jurisdicitions, or
perhaps even in unknown jurisdictions via Web mixes, the file could not be
removed.

-- digital timestamps and other hashes of the database could be published,
a la Haber and Stornetta's Surety service, to ensure that that the database
had not been tampered with.

-- secure data havens, such as Swiss banks (maybe not so secure, but you
get the point) could store copies of the files, perhaps on very slow media.
Enough to ensure eternal storage. Underground vaults, salt mines, the usual
shtick.

I'll close for now.

Could "Eternity.com" be the new Westlaw? (For you nonlawyers, Westlaw
publishes books of court cases and rulings, and is the de facto place for
references...they make a tidy profit by licensing their system. Cyberspace
legal thinkers are looking at Web alternatives.)

Food for thought (and grounds for further research, as Dave Emory would say).


--Tim May

We got computers, we're tapping phone lines, we know that that ain't allowed.
---------:---------:---------:---------:---------:---------:---------:----
Timothy C. May              | Crypto Anarchy: encryption, digital money,
tcmay@got.net  408-728-0152 | anonymous networks, digital pseudonyms, zero
W.A.S.T.E.: Corralitos, CA  | knowledge, reputations, information markets,
Higher Power: 2^756839 - 1  | black markets, collapse of governments.
"National borders aren't even speed bumps on the information superhighway."