Venturing further down the Comcast Netflix rabbit hole

After a few days of contemplation, I've come up with a few topics to cover that will hopefully make a worthy follow up to my last article. The response that I've gotten has been completely unprecedented. The article got over 30,000 hits, over 4,000 shares on Facebook, and I've received countless e-mails and messages over social media offering suggestions, support, and critique. I say unprecedented because this blog usually gets a trickle of a few hits a day. I was invited to speak on a radio station in San Francisco as well as a TV station in Seattle. It seems that people nationwide are quite serious about their Netflix.

Why articles bashing ISPs like Comcast are so popular

Being pissed off at Comcast (or any ISP for that matter) strikes a chord with such a wide demographic of people that it may even encompass every single American who uses the Internet. Americans are mad as hell about their subpar Internet infrastructure, and they're not gonna take it any more. A quick Google search for "comcast netflix slow" turns up thousands of angry forum posts, news articles bashing Comcast, and ranting blog posts about how Xfinity Streaming works perfectly, yet Netflix buffers every 5 seconds.

Since tech support at Comcast is pretty much worthless, and consumers often don't have another viable ISP option in town, all I can do is raise awareness. That's what I plan to do in this article - examine in depth where our Netflix problem may lie and support it with some more evidence.

First, let's make clear what Comcast is not doing

After my last article became popular, I received some criticism from people who took issue with my assertion that Comcast was purposefully blocking Netflix traffic. These naysayers are indeed correct: because of an Open Internet agreement that Comcast inked with the FCC as a stipulation of it's NBC Universal merger, Comcast is prohibited from blocking certain types of Internet traffic until 2018. So, despite the recent Net Neutrality court case, Comcast would be violating Federal Law and violating the terms of their NBC Universal merger agreement if they blocked Netflix.

If Comcast isn't blocking Netflix, then why is Netflix so slow?

The thing is, the FCC Open Internet rules are quite lenient, and make vague exemptions for "reasonable network management". The official text (PDF) of the order says that the "open Internet rules be subject to reasonable network management, consisting of 'reasonable practices employed by a provider of broadband Internet access service to: (1) reduce or mitigate the effects of congestion on its network or to address quality-of-service concerns...'" Because of concerns about vague language in these FCC orders, they were even referred to as "net neutrality lite" at the time of their original introduction in 2010.

Given the high bitrate that Netflix HD requires, and the fact that streaming video is throttled by just about every ISP at some point as part of a quality of service scheme, I think it's totally plausible that Comcast throttles Netflix to some degree. This would especially be true in cases where a neighborhood line is oversubscribed. But I don't think that is the issue that is causing a 240Kbps bitrate on my 50Mbps connection, since many other services work fine. I think the big issue here is peering.

Expanding on the first VPN experiment

For me to make my point about peering, I'm going to present a little data first. When I did my first experiment, it was a simple test that only proved these things:

1. The Comcast connection between my house and Netflix is horrible
2. The Indiana University connection between campus and Netflix is great

Since I don't have complete knowledge of the inner workings of Comcast's network or Netflix's CDN, I may get into a bit of conjecture here, but bear with me. I think we can safely assume that there is a congestion or bottleneck issue at some point in the path data is taking between my Comcast Internet connection and Netflix. Since this problem occurs during off-peak hours as well as prime time, and customers of other local ISPs like U-Verse have no problems, I think we can also rule out Netflix's data center being unable to serve up the data fast enough. At this point, I opened up the Wireshark packet analysis tool to find the IP address of the Netflix node I was streaming from each time, and did a simple traceroute command to see what path traffic was taking from my house to Netflix on the Comcast network:

It's always interesting to watch the various "hops" that packets take as they make their way across the Internet. In this case, we can see that traffic goes north along the Comcast network before hitting the Level 3 fiber backbone, then going into the Netflix data center. Level 3 Communications is a Tier 1 Internet network - one of the major Internet backbone providers that own fiber optic cables criss-crossing the country. Almost all Internet traffic is routed along a Tier 1 network at some point. Additionally, Level 3 is a main provider in Netflix's content delivery network. They agree to host and serve Netflix's movie and TV collection at various points across the country, in exchange for a hefty sum of money from Netflix.

Comcast and Level 3 have a bit of a history

Given the Internet's decentralized nature, it has evolved into a mishmash of various networks owned by companies joined at the edges. When traffic moves from network to network, an interconnect agreement known as peering comes into play. Peering has historically referred to an informal agreement where the two participants are treated as equals - each company sends a roughly equal amount of traffic onto the other one's network, so there are no charges either way. Each company makes money from subscribers on their own side of the larger network. When streaming video began to blow up, many of these agreements lost their symmetrical nature, and companies like Comcast tried to charge backbone providers like Level 3 for access to their network. This Recode article is a great write-up with more information on how the backroom politics of peering could prove to be a bigger issue than Net Neutrality ever was over time.

Essentially, ever since Level 3 inked a deal to be a part of Netflix's CDN, Comcast got a little salty with their old peering peer. Level 3 claims that "once Level 3 signed a contract to deliver video content for Netflix, Comcast jacked up charges for access to its subscribers." The dispute has been running for years, but I think it may have flared up once again.

My hypothesis

Given Comcast's long running fight with Level 3 about peering agreements, and the fact that Level 3 is also a Netflix CDN provider, I believe that Comcast is refusing to pay for the amount of bandwidth needed to handle the amount of Netflix data its customers are streaming from Level 3 CDN nodes.

Proof beyond a reasonable doubt

Unfortunately, there's really no way of me proving this theory. Without knowledge of the inner workings of Comcast and Level 3's networks, I don't have the evidence to prove what Comcast is doing. Peering agreements are hammered out in backroom meetings between Internet providers, and only occasionally see the public light, if the FCC gets involved.

If proof beyond a reasonable doubt is unachievable, then motive is our next best bet. And the motive can definitely be seen here. First of all, Comcast is saving money by skimping on their peering connection with Level 3. By letting traffic pile up at this bottleneck between Comcast customers and Netflix video being served at a node, they're putting pressure on Level 3 to negotiate a better peering price, while their customers suffer. Additionally, as a vertically integrated monopoly, Comcast has no incentive to fix this problem in a timely fashion, or at all. Since Comcast owns NBC-Universal, has incredible negotiating power with its cable content providers, and offers its own Xfinity streaming service, there's no reason why they would put any priority on improving the quality of a service that's only going to poach more of their cable customers over time. And like I said in my last article, since 3/4 of American households only have one choice for very fast broadband, their customers can't do much about it besides complain.

It may get worse before it gets better

As I'm sure anyone reading this article knows, a few days ago Comcast made a bid to buy out Time Warner Cable, the 2nd biggest cable company in the U.S. behind Comcast. Even though Comcast has pledged to shed a few million subscribers to assuage the FCC, consumer advocates are fuming, and many are already saying it will be the death of Netflix as we know it. Many agree it is not going to be a good development for consumers if it does pass regulatory approval, leading to less consumer choice and further dominance of the consumer broadband market nationwide by a single company. And in a depressing reminder of just how little the threat of competition affects Comcast, executive VP David Cohen had this to say:

“The impact on customer bills is always hard to quantify. We’re certainly not promising that customer bills are going to go down or even increase less rapidly.”

It's hard to imagine a merger in just about any other industry where a resulting price increase wouldn't come across as a bad thing. But in Cableland USA, where lobbyists fight to make municipal broadband illegal, real competition is just not a big worry.

Conclusion

When Comcast bought out NBC Universal, they agreed to abide by the FCC's Open Internet rules enforcing certain tenets of Net Neutrality. While the recent Federal court ruling struck down these rules for most ISPs, Comcast is still bound to them until January 2018. Yet because of the backroom nature of Internet provider peering negotiations and the inherent legal wiggle room in the FCC's Open Internet rules, Comcast can allow certain network exchanges to get clogged up with congestion and still be in compliance with these rules. Using the example of the Netflix problem I'm currently dealing with, they may be doing it to pressure Level 3 into a better peering deal, or they may be doing it simply to bleed unhappy customers away from Netflix. It's a win-win for them: Comcast doesn't have to carry quite as much HD video traffic on their network, and they may also gain some new cable subscribers, since Netflix is a must-have for most cord cutters.

Without either regulation or the threat of more nimble competitors, Comcast and other ISPs run the risk of causing serious congestion problems on the Internet due to unresolved peering disputes. In this case, it may actually be beneficial to Comcast to let the traffic jam continue, especially when the cars stuck in the slow lane belong to a major content competitor. Our worst fears about Net Neutrality may soon come to pass, especially when January 2018 rolls around. As always, feel free to let me know your opinion on this issue via e-mail or Twitter.