From: "E. ALLEN SMITH" <EALLENSMITH@ocelot.Rutgers.EDU>
Date: Sun, 5 May 1996 14:29:43 +0800
To: cypherpunks@toad.com
Subject: Tracking Internet Infrastructure
Message-ID: <01I4BK6JR82Y8Y53GG@mbcl.rutgers.edu>
MIME-Version: 1.0
Content-Type: text/plain


From:	IN%"rre@weber.ucsd.edu" 14-APR-1996 07:00:37.86
From: Phil Agre <pagre@weber.ucsd.edu>

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
This message was forwarded through the Red Rock Eater News Service (RRE).
Send any replies to the original author, listed in the From: field below.
You are welcome to send the message along to others but please do not use
the "redirect" command.  For information on RRE, including instructions
for (un)subscribing, send an empty message to  rre-help@weber.ucsd.edu
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Date: Fri, 12 Apr 1996 22:35:54 -0400 (EDT)
From: Gordon Cook <gcook@tigger.jvnc.net>


Tracking Internet Infrastructure:
A Handbook on Business, Technology & Structural Issues Reshaping the 
Landscape of the Commercial Internet 

An Anthology of Recent Articles  from The COOK Report


The Internet has undergone huge changes in the year since the NSFnet 
backbone service was turned off.  It has become a much larger, more 
stratified, and more expensive entity within which to operate.  Since 
last September we have published a series of articles examining these 
changes in depth.  We have concluded that it might be useful to gather 
them together into a single volume which is titled Tracking Internet 
Infrastructure: an organized, indexed Handbook on Internet Infrastructure 
Issues.

This Handbook covers the following critical range of issues:

* interviews with key industry players about viable Internet business 
models, 

* the hierarchical organization of ISPs through CIDR and routing 
pressures, 

* to the viability of the NAPS; 

* to renumbering and ownership of IP addresses; 

* to strains on backbone routers and backbone network redesign with 
switched cores; 

* to industry views on quality of service issues; to issues of 
settlements and route charging; 

* to issues of bandwidth availability in network design; and to ATM as 
technology savior or dinosaur.  

The rest of this message contains:  (1) Our introduction to the Handbook
				(2)  The Handbook table of contents
				(3)  Description of the Handbook's audience
				(4)  Price and ordering information

1.  Introduction

A Summary of the 
Operational 
Environment 

Power Consolidates at the Top of a 
Hierarchical Internet

Less than three years ago, at about one twentieth its current size, the 
topology of the Internet was relatively flat.  Service providers could 
attach to each other via the NSFnet backbone or CIX router and, for the 
most part, they could consider themselves plugged directly into the 
Internet with no one up stream of them.  This meant that no one was in a 
position to dictate to them a multiplicity of rules, regulations and 
costs as part of providing Internet service.

One paid the leased line costs for one's own backbone and, in the case of 
research and education, was connected to the NSFnet backbone for free.  
The R&E networks paid ANS a transit fee for their commercial customers.  
If the service provider were commercial, it joined the Commercial 
Internet Exchange and interconnected at the CIX router for the princely 
sum of $45,000 a year ($10,000 membership fee, $5,000 port fee and about 
$20,000 for T-1 line to the CIX.  Or one was a downstream customer of 
Sprint and relied on Sprint to deliver one's packets to the CIX router 
without having to pay the CIX membership fee.  

Over the period of about a year Sprint, by allowing ISPs to resell 
connectivity, and by giving all ISPs downstream of it connectivity to the 
rest of the Internet Universe, created a situation where customers of 
Sprint received most of the benefits given those ISPs that connected 
directly to the CIX router.  Some ISPs were shocked in mid summer of 1994 
when the CIX proposed that those packets of SprintUs resellers who didn't 
pay the CIX fee would be blocked.

Since then, as the Internet has grown by more than an order of magnitude 
in size, the importance of the NSFnet and CIX interconnects has either 
disappeared or faded.  A very hierarchical Internet has emerged.  One can 
have several levels of upstream service providers.  Such service 
provision ends at a traffic exchange point known either as a NAP or a 
MAE.  Address space in the form of IP numbers is no longer handed out the 
interNIC to all ISPs.  

To get address space direct from the interNIC, you have to do one of two 
things: (1) show that you have no one up stream.  The only way to do that 
is to directly connect to a NAP or MAE and preferably to more than one.  
This level of the hierarchy is reached by perhaps 40 of about 2,400 ISPs 
nation wide.  Or (2) an ISP can multi-home (take a connection from two 
different backbone providers).  Perhaps four to five hundred ISPs are 
multi-homed.  But even a multi-homed ISP is unlikely to get interNIC 
address space, unless it can demonstrate a rapid rate of growth.

Connecting at a NAP or MAE, in an effort to put one's own operating 
environment under one's direct control, is very expensive.  Unlike the 
$45,000 CIX fee the minimum annual cost is $100,000 and up.  Once there, 
providers have to pay additional sums of money to those still higher in 
the hierarchy to see that their packets are delivered.  These sums of 
money are known as transit fees.  Multi-homing is obviously much less 
expensive.

But even at traffic exchange points (NAPs or MAEs) there are additional 
hierarchies.  Some find others who will peer with them.  That is to say, 
they will engage in cost-free transit for a certain percentage of those 
attached.  Those at the peak of the hierarchy are the six service 
providers who are believed to engage in cost free peering and transit 
with each other - MCI, SPRINT, UUNET, ANS, PSI and AGIS.  These six 
operate the default free core of the American Internet.  Exchange of 
routing information among them is supposed to be complete, so that none 
need say: if your routers don't have address XYZ, send packets by default 
to the next large central player, in the hope that his routers will 
know.  Virtually everyone else is, in one sense or another, a customer of 
these top six.  BBN is a special case.  MCI carries all of BBN's routes.  
Consequently in this sense  BBN which is as large as UUNET and certainly 
much larger than AGIS, is a customer of MCI.  We expect this to change by 
year's end when AT&T fully deploys the network for its partnership with 
BBN.  [Editor's Note:  when we fact checked this assertion with sources 
at BBN we were told that BBN is putting its own national T-3 backbone in 
place, that significant parts are now operational, that it is at MAE 
East, MAE West and the Sprint NAP, that it peers with PSI, UUNET, ANS and 
others, and that its transit relationship with MCI is to reach MCI 
customers and any Internet sites not directly reachable via its peers.]

Those who are able to buy transit at a NAP gain by this action gain  the 
ability to ensure it to their downstream (ISP) customers.  In other words 
transit rights are generally resellable.  In this sense, those who 
purchase transit at the NAPs or MAEs are as fully connected to the core 
Internet as the big six - with one critical exception.  They are renting 
their core connectivity for a hefty monthly price and without the 
protection of a long term lease.  Readers however should avoid 
generalizations.  Transit agreements are very private, never talked about 
openly and vary widely on a case-by-case basis.

IP Numbers and Other Indicators of Hierarchy

Another way of describing the top of the hierarchy is to point out that 
all six get their IP numbers directly from the interNIC and hand out 
numbers lower on down in the IP hierarchy to those connected to them.  
But this hierarchy is not uniformly rigid.  The rule of thumb of your 
upstream provider as a source of IP numbers has some notable exceptions.  
For example: of the remaining approximately 35 providers which are 
directly connected to one or more of the major NAPs or MAEs, most get 
some of their IP numbers direct from the interNIC while others are 
derived from being attached at some point or in some way to one or more 
of the central seven.  Finally, as we indicated above, the several 
hundred ISPs who are multi-homed also get their numbers from the 
interNIC, as may ISPs who can demonstrate extremely rapid downstream growth.

An ISP without IP addresses is worthless, since the only thing making it 
possible to connect an individual to the Internet is the IP number that 
tells others how to find him.  The hierarchy of the Internet is now such 
that about 80% of service providers must get their addresses directly 
from their backbone vendors who are often also their competitors.  These 
address blocks are referred to as CIDR blocks.  We have recently written 
extensively about them. 

Until the fall of 1995 large ISPs were able to connect to one or two NAPs 
and sometimes negotiate cost free peering.  As an option they could then 
buy transit to the other five from one of the big six.  During the past 
90 days this has changed.  Reports reaching us indicate that cost free 
peering is available only to those connecting at T-3 speed to three or 
more NAPS a task that will cost an ISP well over $300,000 a year.  But 
even those who do this will find that there is nothing that will force 
everyone at each NAP to which they connect to peer with them.  If one or 
more of the majors refuses to peer with a newcomer, that newcomer will 
have to buy transit to that major from one of those with which it peers 
or be in the awkward position of being unable to reach a significant part 
of the Internet.

For most of 1995 an ISP connecting to a NAP could by transit for $5 to 15 
thousand dollars a month from one of the seven.  But reports reaching us 
now indicate that the six have effectively eliminated the purchase of 
transit from them as an option for domestic ISPs.  These developments 
effectively shut off connection to a major NAP as a means for an ISP to 
operate from the top of the Internet hierarchy.  The only ISPs that may 
even attempt to do so now are those with upwards of $500,000 a year to 
spend on the adventure.  In the meantime the NAPs themselves are 
developing a hierarchy.  MAEs either are opening or have opened in Los 
Angeles, Texas, Chicago and New York.  Without the presence of at least 
the big six at these NAPs, all they are good for is exchanging local 
traffic among local ISPs and keeping loads on long haul backbones down.

The Issue of IP 
Portability

When a customer changes phone service from AT&T to MCI, or visa versa, 
that customer does not have to change phone numbers.  Although the 
portability of  IP address assignments from CIDR blocks has been 
discouraged it has never been prohibited.  But last month a new internet 
draft (draft-ietf-cidrd-addr-ownership-07.txt) by Yakov Rekhter and Tony 
Li of Cisco was published.  This draft created much debate in mid 
February when it was put forth by the CIDRD Working Group for elevation 
to best current practice.  Such elevation would put IETF approval behind 
the practice of a service provider insisting on the return of address 
space when a customer left.  If such a customer were to go to a different 
service provider for a new connection, every one attached to that 
customer's network would be forced to renumber their own networks.  For a 
network of any size renumbering would be expensive and, if that network 
were involved in anything approaching a mission critical application, 
would become unthinkable.

This would very likely mean that any customers buying leased lines to 
connect a network larger than a few dozen hosts in size will find 
themselves well-advised to purchase only from an ISP directly attached to 
one or more of the MAEs/NAPs -- and, therefore, in direct control of its 
address space.  Furthermore, the safest and most conservative action with 
be to connect to the six providers who are part of the default free 
backbone.  Certainly we suspect that the auto industry would tell its 
suppliers not to connect outside the direct NAP connected top tier.

While there are technical reasons for this policy (fear of the collapse 
of internet routing if it is not carried out), it is ironic that such 
policy would greatly accelerate the Internet's stratification into a 
business service and a consumer service, for those who are there to 
explore, just for the fun of it.  There is also an anti-competitive 
aspect to implementing such policy, in that large organization customers, 
which embed non-transferable IP addresses into their network hosts are 
really locking themselves to a single provider.  Should a provider's 
service becomes "less than optimal," we are sure that providers are aware 
that the cost associated with renumbering in order to change vendors, 
limits their customer's options.  While network address translation 
devices (NATs) do exist and will give some customers an alternative, they 
are by no means regarded as a perfect answer to these difficult problems. 

It is beginning to appear that, the more the Internet increases in size, 
the faster that power flows upwards into the hands of a few who, since 
they are both operators and rule makers for the commercial Internet, 
would find themselves singled out for accusations of blatant conflict of 
interest in most other situations.  Under these conditions where the fox 
is essentially in charge of the hen house.  Given the nature of a large 
portion of the customer base (ie large industry and educational 
networks), we wonder how long customers will suffer these burdens without 
demanding regulatory relief.

2.  Contents:  
	Tracking Internet Infrastructure:

	Editor's Introduction
	A Summary of the Operational Environment 		p.  1 

Part One: Internet Business Models
	Some Large Providers Seek Forum to Push for Internet 
	Service Model Change	(Sept.  95)			p.  8

	Interview with Vint Cerf:  Discussion Needed of Benefits 
	Derived from Backbone Resources (Sept.95)		p.  12

	PSI Satisfied with Cooperative Best Effort Internet Business 
	Model  Interview with Bill Schrader(Oct.  95)		p.  17

	Thoughts on Internet Business Models  by Sean Doran   (Oct.  95)
								p.  20

	Zero Sum Internet Business Models Vie with Internet 
	Cooperative Culture	(Oct.  95)			p.  24

	Routing Arbiter & Charging for Routing Announcements: Potential
 	Operational and Financial Impacts Assessed  (Jan. 96)	p.  33

Part Two:  Internet Architecture Change & Network Stratification

	Evolution in CIDR Rules In 1995 Makes Most IP Numbers non
 	Transportable	(Sept.  95)				p.  43

	Constraints of Growth: Provider Based CIDR Likely to Impede 
	Smaller Players   Interview with Dave Crocker and Noel
	Chiappa	(Nov.  95)					p.  46

	Pace of Internet Stratification Increases -- IETF Internet Draft
 	Suggests That Customer Network Renumbering Be Accepted As 
	"Best Current Practice" (Mar.  96)			p.  54

Part Three:  Backbone Routing Versus Switching

	Continued Exponential Growth Stresses Internet Backbone Routing
 	Infrastructure	(Dec.  95)				p.  62

Part Four: Institutions - Sprint; IETF, ISOC, and  the NAPs 

	SprintLink Experiencing Employee Attrition - Executives Slow to
	Provide Staffing Resources Needed for Continued Major Growth 
	(Sept.  95)						p.  81

	National Science Foundation Domain Name Charges Financial
 	Implications for Network Solutions NSF Rationale Behind Actions
	Interview with Don Mitchell	(Oct.  95)		p.  85

	Internet Society:  Role of Charter Members a Contentious Issue
	(Nov.  95)						p.  87

	Transition Pains at the Internet Society (Feb.  96)	p.  88

	Interview with Paul Mockapetris Who Considers Future of IETF &
 	Finds Software Patents a Growing Obstacle (Jan.  96) 	p.  89

	Interview with Tony Rutkowski Who Finds Internet International
 	Coordinating Group Desirable	(Mar.  96)		p.  93

	No Room at Sprint's Pennsauken NAP  (Jan.  96)		p.  96

Part Five:  Quality of Service

	Automotive Industry Will Seek Internet Service Provider
	Certification  (Feb.  96)			p.  102

	Steve Wolff Sees Convergence Between Internet and Telephony
	(Feb.  96)					p.  107

Part Six: ATM and the Technology of Bandwidth on Demand

	Can Bandwidth Supply Keep Pace With Demand? ATM to the Rescue?  
	An Introduction to a Series of Articles on Role of ATM in the Internet
 	(Mar.  96) 					p.  111

	ATM: Grand Unifying Technology or  Brain-Damaged Transport
 	Product?  (April 96)				p.  114

	Bandwidth & Resource Reservation as Factors in Ones Network
 	Provisioning Philosophy -- Can Bandwidth Ever Be Too Cheap to
 	Meter?	(April 96)				p.  117

	Interview with Bellcore's Dave Sincoskie Who Discusses the Internet
 	Future of ATM & Outlines BellcoreUs Interest in Building Network of
 	Interconnected ATM  NAPs	(March 96) 	p.  120

	InternetMCI Bets its Future on ATM  Data Services Marketing & Data
	Services Engineering Vice Presidents Explain MCI Strategy. 
	Interview with Stephen von Rump and Steve Tabaska
	(April 96)					p.  124

	Interview with BBN's John Curran:  Has the Internet Derailed ATM?
	(May 96)					p.  132

Index							p.  140

3.  The Audience for the Handbook

Within the national Internet service provider community, Tracking 
Internet Infrastructure is intended to educate strategists with the 
complexities facing their engineering and operations staff.  Among 
smaller ISPs it should serve as a tool to bring owner-operators, who are 
busy 18 hours a day ordering lines, installing them and servicing their 
customers, up to speed on the changes going on in the environment in 
which they must operate.  LECs and other phone companies will find it 
useful.  Finally familiarity with the issues discussed within the 
Handbook will provide corporate MIS people with a valuable knowledge base 
from which to negotiate with their present or future internet service 
providers.

However, since these infrastructure issues are also critical to the 
continued growth and success of the industry, this Handbook is expected 
to be a tool for use by those in the banking and investment community.  
If those in the financial community understand the changing technical and 
power relationships in the industry, they will be able to improve the 
quality of their investment decision making.  It should also be useful to 
corporate strategic planners who will be advising their companies' 
decision making in vertical industry applications.

4.  This handbook may be purchased in several ways:

A.  Single Copy GBC Bound, double sided xeroxed. $275.00

B.  Site license:  Set of single sided original 600 dpi laser written 
pages suitable for purchasing organization to reproduce as many copies as 
it wishes for its employees only.  $750

C.  A current subscriber without a site license may upgrade to a site 
license and pay an additional price of $275 to receive the report with 
full site license privileges.

D.  A current subscriber with a $650 site license or higher may purchase 
the report with full site license privileges for $275.

To order contact Gordon Cook by phone (609) 882-2572 or email:  
cook@cookreport.com.

*********************************************************************
Gordon Cook, Editor & Publisher    Subscriptions: Individ-ascii   $85
The COOK Report on Internet                 Individ. hard copy   $150
431 Greenway Ave, Ewing, NJ 08618 USA       Small Corp & Gov't   $200
(609) 882-2572                              Corporate            $350
Internet: cook@cookreport.com               Corporate Site Lic.  $650
http://pobox.com/cook/  for new COOK Report Glossary of Internet terms 
*********************************************************************