From: Gordon Cook <gcook@tigger.jvnc.net>
Tracking Internet Infrastructure
Gordon Cook
An Anthology of Recent Articles from The COOK Report- A Handbook on Business, Technology & Structural Issues Reshaping the Landscape of the Commercial Internet
[please look at the footer if you are interested in subscribing to
the Cook Report, ed]
The Internet has undergone huge changes in the year since the NSFnet backbone service was turned off. It has become a much larger, more stratified, and more expensive entity within which to operate. Since last September we have published a series of articles examining these changes in depth. We have concluded that it might be useful to gather them together into a single volume which is titled Tracking Internet Infrastructure: an organized, indexed Handbook on Internet Infrastructure Issues.
Power Consolidates at the Top of a Hierarchical Internet
Less than three years ago, at about one twentieth its current size, the topology of the Internet was relatively flat. Service providers could attach to each other via the NSFnet backbone or CIX router and, for the most part, they could consider themselves plugged directly into the Internet with no one up stream of them. This meant that no one was in a position to dictate to them a multiplicity of rules, regulations and costs as part of providing Internet service.
One paid the leased line costs for one's own backbone and, in the case of research and education, was connected to the NSFnet backbone for free. The R&E networks paid ANS a transit fee for their commercial customers. If the service provider were commercial, it joined the Commercial Internet Exchange and interconnected at the CIX router for the princely sum of $45,000 a year ($10,000 membership fee, $5,000 port fee and about $20,000 for T-1 line to the CIX. Or one was a downstream customer of Sprint and relied on Sprint to deliver one's packets to the CIX router without having to pay the CIX membership fee.
Over the period of about a year Sprint, by allowing ISPs to resell connectivity, and by giving all ISPs downstream of it connectivity to the rest of the Internet Universe, created a situation where customers of Sprint received most of the benefits given those ISPs that connected directly to the CIX router. Some ISPs were shocked in mid summer of 1994 when the CIX proposed that those packets of SprintUs resellers who didn't pay the CIX fee would be blocked.
Since then, as the Internet has grown by more than an order of magnitude in size, the importance of the NSFnet and CIX interconnects has either disappeared or faded. A very hierarchical Internet has emerged. One can have several levels of upstream service providers. Such service provision ends at a traffic exchange point known either as a NAP or a MAE. Address space in the form of IP numbers is no longer handed out the interNIC to all ISPs.
To get address space direct from the interNIC, you have to do one of two things: (1) show that you have no one up stream. The only way to do that is to directly connect to a NAP or MAE and preferably to more than one. This level of the hierarchy is reached by perhaps 40 of about 2,400 ISPs nation wide. Or (2) an ISP can multi-home (take a connection from two different backbone providers). Perhaps four to five hundred ISPs are multi-homed. But even a multi-homed ISP is unlikely to get interNIC address space, unless it can demonstrate a rapid rate of growth.
Connecting at a NAP or MAE, in an effort to put one's own operating environment under one's direct control, is very expensive. Unlike the $45,000 CIX fee the minimum annual cost is $100,000 and up. Once there, providers have to pay additional sums of money to those still higher in the hierarchy to see that their packets are delivered. These sums of money are known as transit fees. Multi-homing is obviously much less expensive.
But even at traffic exchange points (NAPs or MAEs) there are additional hierarchies. Some find others who will peer with them. That is to say, they will engage in cost-free transit for a certain percentage of those attached. Those at the peak of the hierarchy are the six service providers who are believed to engage in cost free peering and transit with each other - MCI, SPRINT, UUNET, ANS, PSI and AGIS. These six operate the default free core of the American Internet. Exchange of routing information among them is supposed to be complete, so that none need say: if your routers don't have address XYZ, send packets by default to the next large central player, in the hope that his routers will know. Virtually everyone else is, in one sense or another, a customer of these top six. BBN is a special case. MCI carries all of BBN's routes. Consequently in this sense BBN which is as large as UUNET and certainly much larger than AGIS, is a customer of MCI. We expect this to change by year's end when AT&T fully deploys the network for its partnership with BBN. [Editor's Note: when we fact checked this assertion with sources at BBN we were told that BBN is putting its own national T-3 backbone in place, that significant parts are now operational, that it is at MAE East, MAE West and the Sprint NAP, that it peers with PSI, UUNET, ANS and others, and that its transit relationship with MCI is to reach MCI customers and any Internet sites not directly reachable via its peers.]
Those who are able to buy transit at a NAP gain by this action gain the ability to ensure it to their downstream (ISP) customers. In other words transit rights are generally resellable. In this sense, those who purchase transit at the NAPs or MAEs are as fully connected to the core Internet as the big six - with one critical exception. They are renting their core connectivity for a hefty monthly price and without the protection of a long term lease. Readers however should avoid generalizations. Transit agreements are very private, never talked about openly and vary widely on a case-by-case basis.
IP Numbers and Other Indicators of Hierarchy
Another way of describing the top of the hierarchy is to point out that all six get their IP numbers directly from the interNIC and hand out numbers lower on down in the IP hierarchy to those connected to them. But this hierarchy is not uniformly rigid. The rule of thumb of your upstream provider as a source of IP numbers has some notable exceptions. For example: of the remaining approximately 35 providers which are directly connected to one or more of the major NAPs or MAEs, most get some of their IP numbers direct from the interNIC while others are derived from being attached at some point or in some way to one or more of the central seven. Finally, as we indicated above, the several hundred ISPs who are multi-homed also get their numbers from the interNIC, as may ISPs who can demonstrate extremely rapid downstream growth.
An ISP without IP addresses is worthless, since the only thing making it possible to connect an individual to the Internet is the IP number that tells others how to find him. The hierarchy of the Internet is now such that about 80% of service providers must get their addresses directly from their backbone vendors who are often also their competitors. These address blocks are referred to as CIDR blocks. We have recently written extensively about them.
Until the fall of 1995 large ISPs were able to connect to one or two NAPs and sometimes negotiate cost free peering. As an option they could then buy transit to the other five from one of the big six. During the past 90 days this has changed. Reports reaching us indicate that cost free peering is available only to those connecting at T-3 speed to three or more NAPS a task that will cost an ISP well over $300,000 a year. But even those who do this will find that there is nothing that will force everyone at each NAP to which they connect to peer with them. If one or more of the majors refuses to peer with a newcomer, that newcomer will have to buy transit to that major from one of those with which it peers or be in the awkward position of being unable to reach a significant part of the Internet.
For most of 1995 an ISP connecting to a NAP could by transit for $5 to 15 thousand dollars a month from one of the seven. But reports reaching us now indicate that the six have effectively eliminated the purchase of transit from them as an option for domestic ISPs. These developments effectively shut off connection to a major NAP as a means for an ISP to operate from the top of the Internet hierarchy. The only ISPs that may even attempt to do so now are those with upwards of $500,000 a year to spend on the adventure. In the meantime the NAPs themselves are developing a hierarchy. MAEs either are opening or have opened in Los Angeles, Texas, Chicago and New York. Without the presence of at least the big six at these NAPs, all they are good for is exchanging local traffic among local ISPs and keeping loads on long haul backbones down.
The Issue of IP Portability
When a customer changes phone service from AT&T to MCI, or visa versa, that customer does not have to change phone numbers. Although the portability of IP address assignments from CIDR blocks has been discouraged it has never been prohibited. But last month a new internet draft (draft-ietf-cidrd-addr-ownership-07.txt) by Yakov Rekhter and Tony Li of Cisco was published. This draft created much debate in mid February when it was put forth by the CIDRD Working Group for elevation to best current practice. Such elevation would put IETF approval behind the practice of a service provider insisting on the return of address space when a customer left. If such a customer were to go to a different service provider for a new connection, every one attached to that customer's network would be forced to renumber their own networks. For a network of any size renumbering would be expensive and, if that network were involved in anything approaching a mission critical application, would become unthinkable.
This would very likely mean that any customers buying leased lines to connect a network larger than a few dozen hosts in size will find themselves well-advised to purchase only from an ISP directly attached to one or more of the MAEs/NAPs -- and, therefore, in direct control of its address space. Furthermore, the safest and most conservative action with be to connect to the six providers who are part of the default free backbone. Certainly we suspect that the auto industry would tell its suppliers not to connect outside the direct NAP connected top tier.
While there are technical reasons for this policy (fear of the collapse of internet routing if it is not carried out), it is ironic that such policy would greatly accelerate the Internet's stratification into a business service and a consumer service, for those who are there to explore, just for the fun of it. There is also an anti-competitive aspect to implementing such policy, in that large organization customers, which embed non-transferable IP addresses into their network hosts are really locking themselves to a single provider. Should a provider's service becomes "less than optimal," we are sure that providers are aware that the cost associated with renumbering in order to change vendors, limits their customer's options. While network address translation devices (NATs) do exist and will give some customers an alternative, they are by no means regarded as a perfect answer to these difficult problems.
It is beginning to appear that, the more the Internet increases in size, the faster that power flows upwards into the hands of a few who, since they are both operators and rule makers for the commercial Internet, would find themselves singled out for accusations of blatant conflict of interest in most other situations. Under these conditions where the fox is essentially in charge of the hen house. Given the nature of a large portion of the customer base (ie large industry and educational networks), we wonder how long customers will suffer these burdens without demanding regulatory relief.
**********************************************
Gordon Cook, Editor & Publisher
The COOK Report on Internet
431 Greenway Ave, Ewing, NJ 08618 USA
(609) 882-2572
Internet: cook@cookreport.com
http://pobox.com/cook/ for new COOK Report
and Glossary of Internet terms
Subscriptions:
Individ-ascii $85
Individ. hard copy $150
Small Corp & Gov't $200
Corporate$350
Corporate Site Lic. $650
**********************************************