Netflix gets an ally in its fight against Verizon, as Level 3 points the finger of blame
Poor old Verizon; the US telco just can’t seem to get a break in its attempts to deny responsibility for causing streaming slowdowns for Netflix customers. It’s already having to bat off the FCC, and now Level 3 has entered the debate with a stinging report that claims Verizon is deliberately constraining capacity from network providers.
Remember how Netflix got so enraged over the slowdown, and accusations by customers that it was at fault, and so posted special splash screens on their video players (we’ve helpfully posted it above, albeit with some added carnage)?
Verizon didn’t like that, and a war of words ensued. Earlier this month, David Young, VP of Regulatory Affairs at Verizon, published a blog post “dispelling the congestion myth” and suggesting that Netflix themselves are responsible for the streaming slowdowns customers have been seeing.
He appeared to shift the blame to “a few third-party transit providers with limited capacity over connections specifically to be used only for balanced traffic flows.” So it was Netflix’s fault for selecting these third-party providers, and the providers’ fault for not doing enough to improve their networks.
“We regularly negotiate reasonable commercial arrangements with transit providers or content providers to ensure a level of capacity that accommodates their volume of traffic,” said Young. “Such arrangements have been common practice for content delivery networks in the Internet ecosystem for many years, and Netflix is fully capable of taking the necessary and customary steps to ensure that its connections match its traffic volumes.”
He concluded emphatically: “There is no congestion on our network.”
One of these third-party transit providers is Level 3, and it didn’t take kindly to Verizon’s accusations.
Mark Taylor, VP of Content and Media at Level 3, wrote a blog post of his own, slamming Verizon’s logic. Alongside Young’s written comments was a helpful network diagram (above), and it was this that Taylor said provided Verizon’s “accidental mea culpa”.
“The diagram shows a network that has lots of unused capacity at the most busy time of the day,” wrote Taylor. “So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.”
So far, so bad for Verizon. Then comes point two.
He argues that Verizon’s diagram distorts the volume and scale of the third-party providers’ networks and hence capacity. He vehemently denies that there is any congestion in his company’s network (which Verizon illustrates as operating at 100 per cent utilisation), arguing that in fact peak utilisations look very similar to those Verizon show for their own network (between 34 and 65 per cent).
The question he then asks is: why does Verizon blame Level 3 and the other network operators contracted by Netflix?
Because, he believes, the congestion is actually occurring at the interconnect points between Verizon’s network and those of Level 3 and others. And to be more specific, at each interconnection location, between a single router owned by Level 3 and one owned by Verizon, and a handful of 10Gbit/s Ethernet ports connected by simple cables (see his own diagram below). Level 3’s network interconnects with Verizon’s in ten cities; three in Europe and seven in the US.
“The aggregate utilisation of those interconnections in Europe on July 8 was 18 per cent – a region where Verizon does not sell broadband to its customers,” said Taylor. “The utilisation of those interconnections in the US – where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses – was about 100 per cent.”
If Verizon claims its network has plenty of capacity, and so too does Level 3, then the only bottleneck is the interconnect building. The simple solution, says Taylor, is to connect up more of the 10Gbit/s ports on the routers in the US locations. Level 3 wants to do this, but Verizon apparently refuses. So frustrated is Taylor that he has even offered to buy some new Ethernet cable and cards and install them himself!
It’s a view supported by Netflix. Ken Florance, VP of Content Delivery at Netflix, revealed in April that his company had to pay Comcast an interconnection fee to improve the quality of its service over the networks of its transit partners – XO, Cogent and Tata, as well as Level 3.
Meanwhile, we await the results of the FCC’s investigation into possible anti-competitive peering and interconnection activities. Then and probably only then, will we be able to determine who exactly is telling the truth.
But for the millions of Netflix customers, they remain blissfully unaware of the complex network transit issues at play here, and unfortunately end up blaming the wrong people for their streaming issues.