• The results are in for the Umlaut survey
• EE wins, Vodafone close behind and O2 and Three further back 
• But how useful is this system measurement approach for users?

The UK network metrics outfit, Umlaut (formerly P3) has released its annual survey results on the performance the the UK’s mobile networks. Obviously the availability of genuine performance data is a good thing. It theoretically provides users and potential users with some performance data on which to partly base decisions over which operator to go with - costs and service levels and which celebrity is endorsing a given network, obviously being the other factors being taken into consideration.  

But over the years, as the network operators built out their networks and refined the services running across them, the inevitable started to happen. 

The mobile network kit and the handsets got ‘better’ and the mobile network operators steadily ironed out the kinks in network performance filling notspots, improving capacity and sharpening their response to outages and network problems. In fact  the metrics improved to the point where it now seems more difficult to tease out real network system performance differences that could be translated into ‘experience’ differences for users.

Here’s how it works

Umlaut’s Mobile Benchmark in the UK consists of drive tests and walk tests all conducted conducted in November 2019. Test cars zip all over the UK visiting an assortment of cities and towns and a ‘walk test team’ get on trains and walk through urban areas. 

Umlaut reckons its total test area takes in 17 million people (27 per cent of the UK population). It also adds in some crowd testing scores.

When all those numbers are boiled down it turns out that (nearly) all the operators are winners... of something.  As in a 21st century classroom (apparently) there are no failures just differences in success. Each network gets its prize (except for Three, in this case), because scorings on particular measures - for instance, Vodafone’s improved Voice performance -  can be accentuated to provide something the network can work up as part of its marketing activity. There’s always a way.

This is how Umlaut summarises its results (see above) for each operator this year. 

It names EE as the BEST IN TEST operator in the UK. EE is the overall winner in the Umlaut Connect Mobile Benchmark in the UK for 2019. Vodafone ranks second with the most improved score since the 2018 benchmark, O2 takes third place from Three this year.  

EE performed best overall taking the lead in all three categories, an overall grade of very good and a score of 876 out of 1,000 points. Vodafone remains second ranked with an overall grade of good and a score of 839 points. 

In comparison to last year‘s results, Vodafone manages to improve considerably in all categories. The second largest operator, O2, ranks third with an overall grade of satisfactory and a score of 697. O2 ranks third in the data and voice categories and second in the crowd category. The fourth place is held by Three, which achieves a score of 621 and an overall grade of ‘sufficient’, which sounds rather like the report card comment “Could do better”.

A critical look at the scoring categories seems to indicate that, in terms of real ‘user experience’ the difference in performance might be less pronounced than the numbers suggest. For one thing in the chart above the combined scores mount up so that EE and Vodafone seem to be well ahead. But score variances within each category are often not that great, especially between the top two. 

Would the voice users, for instance, really notice the difference between operators as represented by scores which range from 219 to 291 against a maximum of 320? 

I don’t know, but it seems to me that individual experiences are likely to be hugely influenced by where a  user lives, works and commutes, and this probably completely overrides - for good or ill - the relevant operator’s aggregate scoring. 

For instance one Vodafone user I know lives in urban London but until recently had to hang out the bathroom window (he swears) to get a reasonable reception. 

There’s no doubt that these surveys are a good thing as they must have the effect of helping crank  up aggregate performance. But as a specific user guide when choosing the best operator? Probably not. Just a starter for 10.