Blog News, tips & advice from the UK's streaming experts

Should you be using “user experience” as the metric for streaming success?

Should you be using “user experience” as the metric for video streaming success-There’s been a lot written about cord cutters and how over-the-top (OTT) programming is going to replace broadcast platforms like Sky, BT, or Virgin. OTT is the delivery of TV or movies online through (or “over”) another device like a gaming system or smart TV using a service like Netflix.

But is watching your favourite TV show through your Xbox One using Netflix or HBO Go, the same as seeing it on your regular TV? The answer to that question is a matter of opinion. But there is some interesting data out there to say that it isn’t, and that perhaps the OTT phenomenon doesn’t really exist – at least in the UK. And the biggest reason for that may come down to user experience and how that is perceived by consumers, and how it’s measured by the companies that deliver the content.

The reality of OTT

According to some recent data from eMarketer, about 63% of Brits watch web video regularly. And regularly means once a month. So in reality, watching online video isn’t taking too much time away from TV viewing. EMarketer also said that in 2016, TV viewing is the top media consumption activity.

And just to bring the point home that OTT doesn’t appear to be taking over the traditional broadcast format, Sky’s annual report shows a strong performance overall. This article on Forbes digs into the details to prove that services like Netflix, and even Sky’s own Now TV, are not making any real impact into the way most people watch TV and movies.

Now that we’ve established OTT isn’t as big a thing as all the talk might indicate, the question is why?

Some would argue, and I agree, that the user experience just isn’t as good. But the next question is, why is Netflix doing so well? And the follow up to that is: As a content creator, do you need to worry about user experience the same way OTT services do?

The truth about OTT

Before we go any farther, I must confess that I view OTT content all the time. But I’m not a cord-cutter. Ok, I did cut the cord once for a few months. But I tied it back together again because, well, it just wasn’t the same.

Most of the time, the video looked pretty good. But finding the content I wanted wasn’t always easy. Sometimes my daughter’s favourite TV shows were on one service. Mine were on another. Switching back and forth meant more than changing a channel. It meant changing remotes (or controllers in the case of video game systems), changing the TV input, and often finding that we had to wait 24 hours or so for the latest episodes. Rarely was anything shown live at the “usual” time even if we wanted to watch it at the scheduled time.

As nice as it was to watch TV shows cheaply this way, it wasn’t convenient. Even if it was “on demand” for when I wanted to watch it, it wasn’t as easy as just turning on the TV.

So I said the video was pretty good – most of the time. But when it wasn’t good, it was usually pretty bad. Unwatchably bad (I know that’s not an actual word, but it conveys the feeling of how bad it was…). Long buffering times were just the beginning. There were fits and starts. Then there were times the video quality was so low there was no point in watching. Or the audio and the video were out of sync, which could be humorous but mostly was just annoying.

Anyway, you get the idea. In concept it sounded great, but in reality there were plenty of challenges. If you’re looking to save money, you might be willing to live with all those inconveniences. I wasn’t – and my kids definitely weren’t.

Video quality

My rant above was mostly about things besides the actual video quality. An article on discusses the same concepts, but refers to video quality “as the elephant in the room” that everyone is ignoring when it comes to OTT.

The truth is, they say, that online streaming quality still just isn’t as good as broadcast TV. The full experience provided by the decades old cable service providers is referred to as “broadcast quality”. And in a line I absolutely love from the article, it says that “These expectations have become known in the online video industry as ‘broadcast quality’, and achieving them has been somewhat like trying to bag a unicorn.”

There really is no better way of putting it than that.

What Netflix is doing

If anyone is going to figure out how to deliver the broadcast experience online, it’s probably going to be Netflix. They’re the largest online content streaming provider in both the US and the UK (probably in the world) in terms of subscribers and number of available titles.

Their success depends on a number of factors, but one of the largest is being able to deliver any of their titles to viewers at the highest possible quality that suits the given circumstances. As you might imagine, they have teams of people working hard to measure the performance of their streaming and making tweaks to improve it.

Just recently, they announced they have created their own metric to measure video quality. Their blog post on the topic gets quite complicated, but the basic idea is they want a metric that will accurately reflect how people feel about the video quality.

Whether a video looks good or bad is a subjective experience. If you read their blog, you’ll see where they put comparisons of videos that have the same sort of statistics (signal to noise ratios mostly), but it’s obvious which looks “better”. (Hint: It’s the one on the lower right. The two on the left look almost identical even though the bottom one is supposedly a higher quality technically speaking).

The assessment, though, of “better” is subjective because from a technical point of view the measured quality is approximately the same. If you have a minute (especially if you’re an analytical type) take a minute to look at the blog so you can see the effect they are talking about and how they are achieving their goal of predicting how their human viewers will perceive their video.

What you can do for your viewer’s experience

It’s not likely you have a full research team like Netflix, but you can put in place some steps to ensure your viewers have a good experience. Unless you’re trying to be the next Netflix, you don’t need to get anywhere near “broadcast quality”.

But you do want your on demand or live video to play on all devices, and do so without lots of buffering, pixelation, or other artefacts.

One way to ensure your video looks good is to put in place a mini-testing team (or a mega-testing team if you’re an enterprise or company that streams lots of video). Here’s a short list (not all-inclusive) of things you’ll want to do:

  • Know how your audience watches. Are they mostly desktop users, Wi-Fi users, mobile users? If you have limited resources, knowing this can save you a lot of time later on.
  • Understand codecs, players, and their compatibility with different browsers. While the situation is better than it used to be, not all video will play everywhere. But most choices of codecs, players and browsers will work almost everywhere – with some exceptions.
  • Test again. And test again, using different devices, in different network conditions. Make sure the video plays on every device you know it needs to, and that your encoding and/or player can handle varying network conditions.

That’s a lot of work yes, but if video is a critical part of your business, mission or community then it’s worth spending some time getting it right. You don’t want to lose out, or look bad, because buffering annoys the heck out of your audience.

The good news is that once you’ve done this work, it becomes more of a maintenance task over time. Browsers change, new codecs get introduced. Take those on as they happen and it won’t feel so overwhelming.

In the end, your viewers will appreciate the time you took to make it all look so good.

Oliver Burt

Sign Up For Free Account