Mobile network performance’s impact on video streaming
As mobile usage continues to grow, the networks are doing their best to keep up. But not all of them are equal, at least according to a new report from Ofcom. The organisation evaluated the major mobile network operators (MNOs) in five major UK cities. The big news reported on many websites was which MNO came out “the best” in the report. The networks included are EE, O2, Three, and Vodafone (the study excluded operators who lease capacity from these MNOs. Companies like giffgaff for example are not included). The overall winner was EE, but there’s more to be learned from the report.
Network performance plays a big role in being able to stream video, so I’m taking a look at the details from that perspective in this blog.
The main metrics
Ofcom chose to focus on two main metrics in making the overall assessment of which MNO is best. Specifically, they looked at download speed and web browsing performance. Both of these activities are probably the most common activities on a mobile device.
The web browsing performance was tested by measuring load times for the BBC homepage. If the page failed to load in 15 seconds, then it was considered a failed attempt. The test was conducted over a period of time, and the BBC page content varied from test to test. Average load times were all well within the 15 seconds, though none of the networks were 100%. That means all of them had some failed attempts.
Download speeds were tested by downloading files (the method used by most speed testing services). What’s important about downloading is that it directly impacts the quality of streaming you can do. The report goes into great detail about how each service performed in each of the five cities examined, but the short of it is that EE outperformed everyone on average (by quite a bit). But all of the networks had averages over 2 Mbps, which Ofcom considered the minimum for “high-capacity video services”.
The averages aren’t the whole story
After seeing that all the services’ average download speeds are sufficient for good quality video, you may think that’s the end of the story. But while sufficient averages are good, it doesn’t guarantee good performance all the time. In fact, there will always be some portion of time where the network isn’t working so well.
To address this, the report looked at both the distribution of the two metrics above, as well as a few other “activities”. These include upload speeds and response time.
The distributions show how consistent network performance is. For download speeds, all networks had a failure rate of about 3%. That means 3% of the time the files failed to download at all. And they didn’t maintain the minimum 2Mbps all the time either. The best network, EE, was above that minimum 92% of the time. The worst, O2, was only above it 69% of the time. This means on both networks there’s a possibility your video stream will fail or encounter trouble, but it will happen much more on O2 than EE (these percentages are averaged over all cities, but individual city results are available in the report).
So far, all the metrics impact watching streaming video. But the upload metrics are hugely important for anyone who wants to live stream using their mobile on a mobile network. Without sufficient bandwidth to upload, your stream will suffer in quality.
Once again, on average, each of the networks had decent upload speeds. EE came out on top with 20Mbps. Vodafone and O2 were at the bottom with 10Mbps. Unfortunately, the distribution of upload speeds is not included in the report. So we don’t know how often an upload failed or what the slowest and fastest speeds are. For example, did most of EE’s uploads cluster around 20 Mbps or were some at 5 Mbps and others at 30 Mbps so the average came in at 20?
Another factor important for live streaming is response time. This is the time it takes for a request from a device (for a web page, image, stream) to be delivered by the network. It is basically a delay. Most of the time this delay is minimal. But when you are doing something like live streaming, it’s important to have the smallest delay possible. The open connection between your device and the server you’re using to stream needs timely exchanges. If the app you’re using to stream doesn’t hear back from the server in time the stream may “time out”. Then you’re no longer streaming. This same issue comes up when making calls via Skype or other video calling apps (basically a live stream both ways!).
While the report lists the average response times, it doesn’t give any indication on how good these numbers are (Three had the best response times in every city though). It all depends on the device, the app, and the server you’re trying to connect to and the settings on each. Some will have a higher tolerance for delays than others. Skype for instance will pop up a message saying, “Trying to reconnect the call” or something similar. They’ve programmed it to try and work around these type of mobile network issues.
In summary, you’ll probably be safe watching streaming video on all the networks. Though on some you may drop down to SD, or suffer buffering more than others. But if you want to really discover the details based on where you live (if you’re in Cardiff, Edinburgh, Liverpool, London or Norwich), you should look at the full report here. There’s a breakout of each city and how the networks perform on each of the metrics for both 3G and 4G networks.