Netflix and A/B Testing Culture

I myself thought I’d read all the articles possible concerning A/B Testing, it’s importance hollowed out, I stumble upon notes taken by Daniel Frankel of FierceCable reporting on Gabriella Mirabelli’s most recent presentation. It concerns the testing methods peak importance in the multi-channel world.

Today, choice is abundant. Especially in the Entertainment Industry. Attention becomes a scarce commodity. Separate channels build a business model centered around getting and retaining it. Today, we see this is the most basic level of competition.

Say I’m waiting for my lecture to start. My professor is 5 minutes late, as per usual. I’ve got some time to kill. The moment I pull my phone from my pocket, a war is waged. That between gaming platforms, social media services, and video streaming applications; for which my attention is the prize. I suppose I could talk to people, but its 2017. Why would I do that when I have a perfectly good phone.

So, how does A/B testing come into this? How does the Symposeum software facilitate this?

In such a hostile environment as described above, one must have to avoid all potential for mistakes. This is even more important in situations controlled by social media platforms, where content is shared from and found via person to person. More simply put, they want to do anything they can to ‘pull in’ the viewer. Not only get their attention, but to have evoke some from of emotional attachment to the  subject in question. Doing that is more than simple guess work. Netflix is the most recent large organisation to adopt a well-known, but not too well adopted philosophy. ‘Test everything’. Whenever there is an internal discussion concerning a creative decision; stop. Stop and think if you’ve run enough tests to be able to answer this altercation externally — using data that can be easily provided by some A/B testing software and an A/B testing tool.

Netflix is a particularly interesting example: “For example, Netflix will show a test audience two versions of a thumbnail for original series “Orange Is the New Black”: one full of elements, the other only featuring star Tyler Schilling and a few text elements. The latter tested much better” Frankel reports. 

Whilst it is always important to remember that ‘it’s not just about fonts’, in the sense that A/B testing can be used in far more complex ways to gain more useful results [more on that here: http://bit.ly/2koHTnB] . But one must ALWAYS remember that the simple red v. green button idea is still incremental in A/B testing. The little things always matter.

Don’t take this ONE example of this, Facebook have more recently in their own app employed this ‘test everything’ philosophy. I argue that one of the many reasons they are the leading social media network is due to the fact that clearly, most of their creative decision’s are made externally. Here (see right), it shows over the past 8 months or so they tested 12 different bottom tab bars. They are aware of the incremental nature of these small decisions based around the design and information architecture. They understand that these changes must be tested in the real world.

By | 2017-12-12T01:06:17+00:00 December 12th, 2017|Categories: Uncategorized|0 Comments