Wharton Business School professor Kevin Werbach dubs Comcast’s actions this week a “turning point in US Internet policy.” Law professor Susan Crawford calls Comcast a terrifying, hat-wearing hydra—and she’s looking for a Hercules to cut it down to size. Harold Feld of Public Knowledge says that Comcast has set up a new “toll booth” on the ‘Net and is now operating like Ed “use my pipes free” Whitacre. And broadband analyst Dave Burstein says Comcast has just deployed “the nuclear option.”
Just what is going on here, why does it matter, and why is Comcast calling backbone operator Level 3 a big fat liar for starting the whole debate?
The facts as we know them
Comcast found itself in the middle of a renewed argument over its “evilness” yesterday afternoon as an interconnection dispute blew up into public view. Level 3 Communications, which runs a major Internet backbone along with a newer content delivery network (CDN), fired off a scathing press release accusing Comcast of (once again) threatening the “open Internet.”
On November 11, Level 3 inked a deal with Netflix to serve as the streaming media company’s new CDN starting January 1, 2011. In that capacity, Level 3 will cache and serve Netflix streaming video from sites across the country to avoid possible backbone congestion and to deploy streams from servers that are closer to end users.
Due to the size of the deal, Level 3 announced that it was doubling its own storage capacity and adding 2.9 terabits per second (Tbps) of CDN capacity, alongside the 1.65Tbps that it rolled out late this year. The entire Netflix streaming library, which consists of more than 20,000 titles, will be moved directly to Level 3 servers during November and December in preparation for the January rollout.
The deal also means, of course, that a huge new amount of traffic will soon be sent from Level 3 to various ISPs, and not all of them are happy at the prospect. A week after Level 3 announced the Netflix deal, Comcast told the company that it would “demand a recurring fee from Level 3 to transmit Internet online movies and other content to Comcast customers who request such content,” in the words of yesterday’s Level 3 press release.
Comcast says that this is standard practice (and that it’s not singling out any particular kind of content). While it previously had a settlement-free (read: no charge) traffic peering agreement with Level 3, such agreements are typically made only between network operators who are roughly comparable in size and therefore exchange similar volumes of traffic with one another. According to Comcast, the Netflix deal means that Level 3 will soon more than double the amount of traffic it sends onto Comcast’s network and that this will result in a 5:1 traffic ratio—at which point the relationship will be unbalanced, and fees will be required.
“We are happy to maintain a balance, no-cost traffic exchange with Level 3,” said Comcast senior vice president Joe Waz last night. “However, when one provider exploits this type of relationship by pushing the burden of massive traffic growth onto the other provider and its customers, we believe this is not fair.”
Level 3 thought it was entirely fair—in part because this is not traffic that is transiting through Comcast’s network, but traffic that is headed for Comcast subscribers and was requested by them. Under this view, Comcast is trying to charge a content provider for the very access to content that it is selling to its own customers. And, of course, there’s the issue that such fees could raise costs for Netflix and other Internet video providers, but won’t affect Comcast’s own video services such as cable TV.
“By taking this action,” said Level 3 Chief Legal Officer Thomas Stortz, “Comcast is effectively putting up a toll booth at the borders of its broadband Internet access network, enabling it to unilaterally decide how much to charge for content which competes with its own cable TV and Xfinity delivered content. This action by Comcast threatens the open Internet and is a clear abuse of the dominant control Comcast exerts in broadband access markets as the nation’s largest cable provider.”
And if that wasn’t enough of an attack, Level 3 said that it was “approaching regulators and policymakers and asking them to take quick action to ensure that a fair, open and innovative Internet does not become a closed network controlled by few institutions with dominant market power.”
While such sentiments tap into a fertile seedbed of hate that many people have for cable companies in general (routinely one of the most reviled industries in the US by consumers) and for Comcast in particular (thanks to its BitTorrent blocking/throttling and subsequent litigation), Level 3 is hardly a disinterested paladin working for Great Justice; the company wants to maintain its free peering arrangements with Comcast even as it grows into a large CDN operator.
This appears to be Comcast’s view of the situation—it’s a private peering dispute taken public in a bid for leverage. And Level 3, seeing no choice if it hoped to avoid a service interruption, has already agreed “under protest” to the new fees.
So is there something new here?
The hydra grows a new head
The details of peering and transit are notoriously tricky to unearth (though our comprehensive guide to the subject helps), since most are locked up behind private deals and nondisclosure agreements, but several people we spoke with believe this to be an unusual stance taken by Comcast.
“As far as I know, no primary backbone provider like Level 3 has ever been required to pay to deliver traffic to another major carrier,” says industry expert Dave Burstein. “Comcast’s explanation why they should collect from Level 3 would also apply to every other backbone carrier and hence to essentially every bit carried over the Internet. Downstream traffic is about four times as high as user upstream, so even a company with a lot of users receiving packets from Comcast users would be asymmetrical.”
Major last-mile ISPs, including AT&T and Verizon, also have these imbalances when connecting to major backbone providers, since (despite “Web 2.0” tech) users still download far more content than they upload. Indeed, Comcast’s own statement makes clear that Level 3 has long been sending much more traffic to Comcast than the other way round.
To Burstein, Comcast’s move is about shifting to a world where every bit is paid for. The problem, as he sees it, is that there’s not enough competition and so the big last-mile ISPs would be in a perfect position to set unreasonably high rates, improving their own profits and blocking out competitors like online video. Why bother to block traffic, which would raise an outcry, when you can simply tax it to death in private?
“Flow through to consumer benefits would be plausible with strong competition,” Burstein tells Ars, “but with weak competition I follow almost all the money straight to carrier bottom lines.”
Rob Frieden, a Penn State telecoms prof and recent author of the book Winning the Silicon Sweepstakes, has studied such issues for years. He says that Comcast’s argument about this being a mere peering traffic imbalance is plausible, but he notes that the company could do plenty to adjust its own routing strategies to send more off-network traffic through the Level 3 backbone in order to make the ratios more even. But Comcast might well have an incentive not to do this, “deliberately reducing the volume of ‘return’ traffic it hands off” in order to make the imbalance even larger.
The end goal, of course, would be getting Level 3 to pay up and move away from free peering.
“It strikes me as a little too easy and expedient for Comcast to resort to the Level 3 surcharge model,” says Frieden. “Bear in mind also that when Comcast charges $50+ for broadband—and recently raised its rates—a company has to expect that its subscribers expect payment to include access to Netflix and other sources of full-motion video. Why would it be okay to download hundreds of gigabytes per month, provided the upstream peer is not Level 3? This does not fully pass the smell test in light of Comcast’s ulterior motive to raise the cost of a competitor, Netflix, and its preferred carrier.”
Due to the general secrecy around these kinds of deals, no one we spoke with could say exactly what was at stake in this case, but suspicion of Comcast is certainly widespread.
Susan Crawford, previously with the Obama administration and now back to teaching law at Cardozo Law School, says the entire battle shows Comcast’s “existing overwhelming market power.” In her view, there’s simply not enough competition to create a functioning market for peering and transit. It’s time for the FCC to act, she says, “as the looming cable monopoly stops looming and starts muscling levers into place.”
Not a peering dispute?
The DC group Public Knowledge blasted Comcast’s stance as a net neutrality violation. Many others, even those who oppose the move, would not paint it quite in those terms, since Comcast is not actually discriminating against certain kinds of bits. As Wharton professor Kevin Werbach tweeted this morning, “The peering issue isn’t net neutrality; it’s whether market forces are failing for interconnection. Seems so.” Werbach attacked the Comcast move instead as a “classic terminating access monopoly case.”
In other words, Comcast is holding its users’ eyeballs hostage, demanding payment for those who want to reach them because there’s no other way to get data to those customers.
(Update: In a follow-up e-mail, Werbach expands on his tweet to note that “terminating access monopoly doesn’t necessarily mean the action is harmful—it means this is a situation where Comcast has effective market power regardless of whether there is broadband competition. So the action should be scrutinized, but that’s not the same as attacking it. We need all the facts to make that determination.”
Though he’s increasingly comfortable with Comcast’s position here, he remains “troubled by the direction this could take the whole backbone market. Which is why it’s appropriate for public policy concern. Most of the time everything will still work itself out best through private negotiation, but it’s unrealistic to believe that will always be the case in the current situation of consolidation and integration across markets.”)
Several Ars staffers have experience in maintaining peering and transit connections both in the US and Europe, and each agreed that the situation here is unusual. That’s because most “transit” deals, the ones where money was exchanged, historically focused on data that was simply traversing one network on its way someplace else. Why should one network operator bear the costs of building and maintaining a network just so that some other network operator could route all of his traffic over it for free? Peering, or direct network interconnection, generally took place when each network sent similar amounts of traffic to the other and it wasn’t worth the expense or hassle of trying to account for every bit.
But the CDN traffic from Level 3 isn’t in “transit” anywhere; it’s going to the Comcast customers who want to watch Netflix movies. Level 3 is, in one sense, doing Comcast a favor by making a key Internet service better; it’s not simply taking advantage of Comcast’s network to get its own traffic somewhere else. That’s what Werbach means when he talks about a “terminating access monopoly”; Comcast has a lock on its customers and can try to extract rents from anyone trying to send them data, even if it’s data they requested.
I spoke with Public Knowledge legal director Harold Feld, who makes the same points about peering and transit. “To the best of my knowledge,” he says, “this is the first time ever a last-mile network has demanded compensation from another ISP for delivering traffic requested by a subscriber on the ‘terminating’ ISP’s network.”
Peering disputes have been around for years, and have sometimes resulted in slow Internet connections as one provider yanks its direct connection to another network, but Feld says that past disputes have “involved hand off of traffic through the middle-mile to transport it from one last-mile network to another.”
In the Level 3 case, “if a last-mile ISP can charge for ‘peering’ with an end-user subscriber that requests the data, how is that different from charging Netflix directly to deliver their traffic?”
Bottom line: this isn’t a simple “peering” dispute. It is instead a “demand to a content provider’s ISP to pay a new fee to deliver content requested by a Comcast subscriber.” In short, it is Ed Whitacre’s dream Internet.
Everybody’s doing it
In its defense, Comcast says that it has similar deals in place with other CDNs—apparently Limelight and Akamai—and that they have signed on to “mutually acceptable commercial agreements… In delivering the same types of traffic to our customers.” (This is known as “paid peering,” in which direct, nonsymmetrical interconnections are allowed for a fee. DrPeering has a nice overview of the practice.)
In fact, Level 3’s choice to cry havoc and let slip the dogs of war is “simply duplicitous,” Comcast says. “When another network provider tried to pass traffic onto Level 3 this way, Level 3 said this is not the way settlement-free peering works in the Internet world. When traffic is way out of balance, Level 3 said, it will assist on a commercially negotiated solution.”
This is in reference to Level 3’s own dispute over peering with Cogent, in which Level 3 eventually discontinued peering with Cogent’s network after reviewing the relationship and concluding that too much traffic was coming from Cogent.
“Cogent was sending far more traffic to the Level 3 network than Level 3 was sending to Cogent’s network,” said Level 3 at the time. “It is important to keep in mind that traffic received by Level 3 in a peering relationship must be moved across Level 3’s network at considerable expense. Simply put, this means that, without paying, Cogent was using far more of Level 3’s network, far more of the time, than the reverse. Following our review, we decided that it was unfair for us to be subsidizing Cogent’s business.”
But note that this was traffic in transit, not traffic that was requested by Level 3 subscribers.
The issue comes at a delicate time for Comcast, something that Level 3 was no doubt aware of. Comcast is currently trying to wrap up its merger with television network NBC and hopes to avoid a host of onerous conditions on the deal. Charges of being an Internet bully won’t go over well at the FCC, where Comcast has few friends after single-handedly gutting the agency’s network neutrality authority by winning a major court case early this year.
Public Knowledge wants the FCC to exert oversight over these types of interconnection agreements, perhaps by setting up some ground rules and offering mediation where the negotiations go bad. Werbach envisions something similar, with more transparency to these negotiations and “perhaps an arbitration process” that would work much as TV retransmission disputes do now.
But if the Comcast/Level 3 dispute really is a turning point in US Internet policy, it’s not yet clear that the government intends to do anything in response. The FCC has one open meeting left in December before the new Congress convenes in January, and if the agency decides to take on network neutrality, this would be the meeting at which it does so. We should know within the next few days if the FCC hopes to act, though it will take much longer to figure out if such hypothetical action would even address the Comcast/Level 3 situation or would focus instead only on traffic “discrimination.”
Asked today if the FCC was examining Level 3’s dispute, Chairman Julius Genachowski would only say the agency was “looking into it.”
- Feature: How Comcast became a toll-collecting, nuke-wielding hydra (arstechnica.com)
- Comcast tells FCC that Level 3’s complaints have no merit (latimesblogs.latimes.com)
- A Play by Play on the Comcast and Level 3 Spat (gigaom.com)
- FCC Looks Into Level 3, Comcast Content Dispute (pcworld.com)
- Comcast, Netflix partner feud over ‘open Internet’ (alternet.org)
- What The Comcast/Level 3 Fracas Is Really About: Money (techcrunch.com)
- Level 3 vs. Comcast: More Than A Peering Spat? (datacenterknowledge.com)
- Comcast, Netflix partner feud over ‘open Internet’ (newsinfo.inquirer.net)
- Netflix is a bandwidth hog. Who will pay? (Hint: You.) (money.cnn.com)
- US cable giant accused of internet video ‘toll booth’ (go.theregister.com)