The Internet is in for interesting times. Previously, I wrote about the engineering issues and about the policy issues facing us over the next five years. But there is at least one large issue still lurking. Most of you will not be surprised to learn that almost all of these issues are outgrowths of a single factor: money. The core of the Internet still doesn't have a sustainable business model.

Many people are getting rich on the Internet, and almost none of them are spending money to keep the interconnection infrastructure (the "Inter" in "Internet") growing and expanding. Look at it from a massively oversimplified perspective: Google make their money from the advertising they sell to search audiences. Comcast make their money by offering TV and Internet access on their local cable infrastructure. Amazon make money selling books and other stuff (including servers and storage space). Most datacenter companies make their money selling space and power inside of their buildings. Spammers make money filling up your inbox with useless crap. Organized crime makes money by launching attacks against profitable companies if they don't pay extortion. DNS squatters make money registering thousands (or millions) of domain names and sitting on them until someone else is willing to pay. And almost none of this helps the core of the Internet.

Look to the wholesale carriers if you want to see an income statement wasteland. Level 3 lost $1.1b last year. They lost $120m in the most recent quarter alone. Cogent is thrilled because they reported a tiny, tiny positive net income last quarter on top of a yearly loss of $30m in 2007. Global crossing lost $300m in 2007 and $88m in the last quarter they're reporting, which doesn't include much of the recent downturn. Other wholesale networks are in the same boat. Dan Golding suggested that it's more important to look at net cash flows rather tha income, but the result is pretty much the same: almost no one is making any money. The only wholesalers who do make money make it on other service offerings: wireless service, metro Ethernet services, VPNs, local phone service, video services and so on. Are there sustainable Internet backbone business models? Does anyone have one?

An open market for buying and selling IPv4 Addresses is coming. Soon. As I wrote previously, IANA is running out of unallocated IPv4 addresses. Estimates vary, but by 2010 (or 2012 at the latest) the world will be out of unallocated IPv4 addresses.

Sometimes it is hard for the general public to understand what this might mean. Essentially, after 2010 or so, if you want to start a new company and get connected to the Internet or just are growing and have more devices that need to have IP addresses, things won't be the same as they are now. Right now what happens is that you go to ARIN, if you're in North America and document your need for IP addresses, you pay a modest administrative fee, and then they allocate them to you. If you grow and you need more, you document how you've used up the ones that you have, and they give you more of them.

All of this assumes that you want your own IP addresses that are not tied to any particular provider (this is an important point that we'll get back to). But even if you get your IP addresses from some provider, they have to get them from somewhere. If you want to be reachable from the Internet, you need an IP address—an IPv4 IP address in particular. And very shortly those are going to get much harder to get.

So let's talk about what happens after the IPv4 addresses are all "used up."

In a few weeks, I will be leaving Renesys, a company I have been associated with for over five years. I moved from New Hampshire (where Renesys is headquartered) to Pittsburgh, PA, over the summer, and I've decided to work a bit closer to my new home.

Before I go, there is work yet to be done. The Renesys blog has become an important place for Internet engineers, managers, developers and salespeople to seek unbiased information about what is happening on the backbones. I have enjoyed contributing to it over the years, and I have enjoyed watching some of my colleagues (most actively Earl Zmijewski and Martin Brown) take the helm more recently. Before I ride off into the sunset, there are at least two things I'd like to contribute to this forum:

1. A clear assessment of where we are with this whole Internet project
2. A good guess about where we're going

At the end of the next series of posts by me, you should either be very, very worried or convinced that I'm very, very wrong. The Internet is facing a confluence of engineering, financial and policy storms that have some small potential to completely derail it. These tempests have a high likelihood of marking a sharp departure from several characteristics once considered fundamental to the the Internet.

If we get through the next five years, I'm sure everything will be fine. Today, I'll tackle the technology and engineering issues. In my next post, I'll address financial issues, followed by policy issues. At the end of this torrent of pessimism, I'll try to point to some plausible ways out of the mess that we have gotten ourselves into.

Sprint re-enabled the connection between Sprint and Cogent at 21:00 UTC (16:00 EST) on Sunday, 2 Nov, 2008. Sprint issued a hastily prepared statement about the reconnection (the HTML is a cut-and-paste job from "IP/MPLS Products from Sprint"), explaining their position. Cogent hasn't commented yet.

The connection appears to be routed much as it was before Oct 30. Previously, we saw Sprint selecting 2700-2900 prefixes from Cogent (that is, picking Cogent as the best path for that many network prefixes). We saw Cogent selecting about 7500-8000 prefixes from Sprint. Now that they have reconnected, Sprint is selecting 2538 prefixes from Cogent and Cogent is selecting 7016 from Sprint. So down slightly, but not appreciably. The link is up.

The fact that Sprint has reconnected this indicates clearly that they intend to fight this battle in court rather than in the routing tables or in the court of public opinion. This fact alone makes this likely to be one of the more interesting peering disputes of the last few years. But the resolution may take months or years, given the speed with which the courts move.

A special Halloween edition of the Renesys Blog: That which was whole is now torn asunder, and cries of grief ring out throughout the land. Cogent (AS174) and Sprint (AS1239) are no longer connected to each other. Customers of each network who do not have other providers—namely single-homed customers—cannot reach each other. Two large portions of the Internet are separated.

Cogent is frequently involved in peering disputes. In the last three years, the only significant peering dispute (one that caused a temporary partition of the Internet) that did not involve Cogent was between Level 3 and XO. That one was settled very quickly. All of the others (Cogent depeering Telia, Level 3 depeers Cogent, and further disputes going back years involving Teleglobe (now Tata, AS6453), France Telecom (AS5511)) involved Cogent.

But in this case, Cogent may have picked the wrong sparring partner. In the past, Cogent won peering disputes simply because their customer base was less sensitive to the outage than the other party in the dispute. Ultimately, the one whose customers complain the loudest loses. This time it may be very different. Sprint hasn't paid any particular attention to its IP product and network at a senior management level for a very long time. They are clearly focused on wireline and wireless telecom services and Overland Park management seem to remain mostly unaware that they even operate an IP network. In other words, Cogent has picked a fight with a zombie here. They may even rip off a limb or two, but that doesn't mean the zombie will notice.

Sprint and Cogent only starting peering recently, back in November of 2006. Prior to that the two networks reached each other via NTT Communications (AS2914). Now, almost exactly two years later, it appears that Sprint has disconnected Cogent and chosen to divide the Internet. Cogent has stated that they will litigate this issue so this one is unlikely to get resolved quickly. In the mean time, over 200 downstream autonomous system customers of each organization cannot reach the networks in the other. This is ugly and will remain so.

Let's take a quick look at what we know so far and set the stage for a story that will likely continue for several days, if not weeks. I'll also try to set this in a larger context about the evolution of each of these networks and the evolution of Internet interconnection on the whole.

Atrivo (aka Intercage), a Concord, California-based Internet hosting service, disappeared from the Internet for around two days recently. They didn't go bankrupt or suffer a physical catastrophe. Their providers simply shut them down by refusing their traffic. This might very well be the first time in history that the Internet community, a cooperative association of networks with no governing body, has collectively put someone out of business, if only briefly. The alleged sins of Atrivo have been documented extensively, both in the popular media (e.g., the Washington Post) and in technical forums (e.g., Spamhaus and numerous postings to the NANOG mailing list). It is clear that emotions run high with respect to Atrivo, long accused of benefiting from cyber-crime by hosting purveyors of malware, adware, spam, viruses and other cyber-surges. In this blog, we'll take a quick look at their brief demise and make a few observations.

Ike swept across Texas on Saturday, devastating Galveston and severely damaging Houston and Harris County before plodding intensely north and east through Arkansas, Missouri, Illinois, Indiana and Ohio. While many Texan networks remain unreachable, the recovery is proceeding remarkably quickly. From a peak of around 100 networks suffering outage through Monday, Ohio, still hard hit, is also slowly starting to recover. And Pennsylvania is the latest victim of Ike.