• Spain and Portugal suffered a major power outage this week
• Telcos scrambled to switch to power back-up options to maintain at least some services
• Ookla and Cloudflare have reported on the performance of the mobile operators and the impact on internet traffic volumes

Network operators of all kinds are increasingly concerned about the impacts of cyberattacks but there are plenty of other security concerns to be considered too – just ask the telecom network and datacentre operators in Spain and Portugal about the impact of a national power outage, something that the Iberian peninsula has just experienced. 

The power grid suffered what is known as “net zero” in the middle of the day on Monday 28 April, leading to major disruption to transport, trade and general life in the two countries and the need for emergency power supply measures in critical sectors, such as healthcare and communications. The power grid was brought back online in increments starting later the same day and, by the middle of Tuesday, 99% of the grid was restored. A cyberattack has, at least so far, been ruled out as the cause of the outage, according to this Reuters report. 

So what impact did this have on communications services and network operations? In the first instance, Telefónica, for example, activated its Crisis Committee and turned to back-up power generators and batteries to keep at least some network systems running. But even a few days after the initial outage there are still issues to be resolved by the major Spanish telco, which says it has  reached 97.5% operational capacity on its fixed network and 95% on its mobile network in Spain.

Vodafone Spain also invoked its Crisis Committee operations and took to social media late on 28 April to note that “all network control centres, such as datacentres and other systems, remain operational and have sufficient autonomy until power is restored”. At noon on the following day, it reported that it had recovered 99.5% mobile network coverage and stated that “virtually all of our fibre and mobile customers can use their services normally”.

Ookla, which monitors network performance key performance indicators (KPIs), provided fantastic insight into the impact of the outage on mobile service performance in Spain and Portugal in this speedily prepared analysis. Using data from its Speedtest Intelligence and Downdetector platforms, Ookla analyst Luke Kehoe noted that “network performance progressively deteriorated as battery backups, where available, were exhausted, and shifting traffic patterns placed maximum strain on an already stretched mobile site grid. The scale of the disruption – with near-synchronous impact across all operators in both countries – marks the largest stress test of telecommunications infrastructure in Europe in recent memory.” 

The impact peaked mid-afternoon on Monday “as backup batteries became depleted. The share of Spanish mobile users experiencing a consistent network connection (defined as minimum 5 Mbit/s download speeds and 1 Mbit/s upload speeds) dropped from a baseline of over 90% at 9am CET on 28 April to 50% by 12pm, reaching a low of 40% by 3pm. The decline in network consistency was even more pronounced on Portuguese mobile networks, falling below 40% by 2pm and initially recovering more slowly than in Spain. Median mobile download speeds on 28 April were 73% lower in Spain and 75% lower in Portugal compared to the previous day,” noted Kehoe. 

While all service providers experienced “substantial service disruptions”,  according to Ookla’s analysis the best performing operators on 28 April were MásOrange (the largest operator by mobile customer numbers) and Movistar (Telefónica) in Spain, and Vodafone in Portugal.

Naturally, digital services infrastructure and service levels were also affected. Cloud connectivity platform Cloudflare noted in this blog that in Spain, internet traffic volumes dropped as the power grid failed, “with traffic immediately dropping by around 60% as compared to the previous week” and “falling to approximately 80% below the previous week within the next five hours. The power outage also impacted the quality of connectivity at a national level in Spain. Prior to the loss of power, median download speeds across the country were around 35 Mbit/s, but within several hours after the state of the outage, fell as low as 19 Mbit/s.”  

In Portugal, internet traffic volumes immediately dropped by about 50% compared to the week before, “falling to approximately 90% below the previous week within the next five hours,” according to Cloudflare. “The power outage also impacted the quality of connectivity at a national level in Portugal. Prior to the loss of power, median download speeds across the country were around 40 Mbit/s, but within several hours after the state of the outage, fell as low as 15 Mbit/s.”  

What seems particularly concerning is that the alternative power sources didn’t sustain reliable operations for very long, highlighting the need for the operator community and industry bodies to focus more of their R&D and crisis management efforts on power management, something that is likely to become even more critical as network operations become increasingly reliant on datacentre-hosted autonomous network management systems, and as ongoing investments in more and more hyperscale datacentre facilities puts an increasing strain on energy infrastructure. 

- Ray Le Maistre, Editorial Director, TelecomTV