Netflix and its rivals are using AI and lessons from Bandersnatch to make shows even more binge-worthy

Samantha Lee/Business Insider

• AI is already a recognizable feature of streaming services through their recommendation functions. The future means more customization, and more ways to use data to identify ways to tap into viewers' moods.
• Netflix is using interactive content, like its recent Bandersnatch special, to provide a feedback loop that will help improve recommendations.
• This article includes an overview of AI in media, plus the top three trends to watch, plus how Disney Streaming Services is using AI to connect with viewers on an emotional level.
• Read how AI is transforming health, transportation, investing, and more in other articles from our special report, How AI is Changing Everything.

When you sit down to watch a TV show or movie on Netflix, or one of its streaming-video competitors, the curation of the sprawling collection of titles you see before you is usually led by algorithms.

Machine learning, a field of artificial intelligence, helps sort through the thousands of titles on platforms like Netflix to determine which to display, in what order, and with what artwork or other features, based on your interests.

Personalization is a key part of how Netflix, Amazon Prime Video, Hulu, and others help viewers find content they want to watch quickly, so that they keep watching and engaging with the platform.

Content recommendations - and the user experience more broadly - are becoming more important at streaming-video platforms as more subscription services hit the market, including the soon-to-launch Disney Plus, Apple TV Plus, WarnerMedia, and NBCUniversal platforms, each of which will have deep libraries of popular, star-studded, or award-winning programming.

"Apple, Netflix, Amazon, Disney, WarnerMedia, Comcast - they will all compete in the user experience," Corey Halverson, vice president of media management at streaming-technology company Akamai, told Business Insider. "That user experience will be largely influenced and perhaps eventually driven by AI."

Recommendation engines are only as good as the metadata that fuels them. Machine-learning systems are doing a lot of work under the hood, in speeding up the monotonous but necessary task of tagging each piece of content in a platform's library with attributes such as genre, actors, major themes, primary languages, or even visual styles, which are used to make recommendations and respond to search queries.

Algorithms tend to make the best use of that metadata when they are prompted by users, with a search for "documentaries about true crime," or by selecting a category like movies or TV dramas.

There are limitations to the recommendations that machine-learning systems can make on their own.

Platforms including Netflix and Amazon make recommendations based on users' viewing patterns, as well as the patterns of people with similar tastes, for example. At what point should the algorithms deviate from those patterns? If you've been binging a lot of sitcoms, should the algorithms attempt to break you from that rut with a drama or feature-length romantic comedy?

A person's interests on a rainy Saturday afternoon may also be very different from a Tuesday night, when they may be looking to unwind.

"Taste or mood is a really difficult thing to predict at this stage in the industry," Jeanine Heck, vice president of Comcast's artificial intelligence product, said. "I don't think AI is there yet where it's able to read your mind."

Netflix's heavy reliance on algorithms has upset some users in the past. Netflix was criticized for racial targeting in 2018 after some black users said Netflix was pushing titles to them using thumbnail images that featured black actors, who in some cases had small parts in the movies or series being promoted.

"We don't ask members for their race, gender or ethnicity so we cannot use this information to personalise their individual Netflix experience," Netflix said in response at the time. "The only information we use is a member's viewing history."

Many content recommendation systems still require some level of human guidance or curation, whether it's through A/B-tested changes to the algorithms, or manual video playlists.

Some media companies are looking for inspiration to platforms outside of video, like the music-streaming service, Spotify, which creates daily playlists of songs for each user, personalized playlists for certain occasions, like songs of the summer, and also has a deep library of other playlists for almost any mood.

Each company, be it Netflix or CBS All Access, is also tailoring its recommendation systems, to a degree, to serve its business needs.

Read more: A WarnerMedia exec describes the challenges of uniting HBO, Turner, and Warner Bros. and launching the company's new innovation lab

WarnerMedia, for example, is using human curation to its advantage over more tech-driven competitors like Netflix and Amazon Prime Video.

"What we have, at WarnerMedia, is some of the best curators and storytellers in the world,"

Jesse Redniss, head of the WarnerMedia Innovation Lab, an incubator arm that is helping to develop technology for the company's upcoming streaming service, said. "Super-charge that with some of the AI capabilities, I think that's what's really going to help push our product to the next level."

Fionna Agomuoh/Business Insider

Top 3 opportunities for AI in media

Metadata: Growing

Improving the metadata underlying the content and drawing insightful connections from the data.

Customization: Nascent

Personalizing the user experience facilitating the content recommendations.

Recommendations: Nascent

Recommending videos based on viewers' tastes, as well as their moods or situations.

Netflix looks for more ways to get viewers binge watching

Netflix, the global giant in streaming video, is one of the companies furthest along in using machine learning to fuel content recommendations based on viewers' tastes, as well as to customize the ways those recommendations are delivered.

Almost every user who logs into Netflix gets a different experience. Netflix's recommendations are personalized to each user account, as are the rows, the title artwork, and the order in which that information is displayed. The company is constantly A/B testing other ways to make recommendations - and entice people to watch more on its service - such as playing trailers in between episodes or changing the visuals associated with each title in the Netflix library.

"This is yet another way Netflix differs from traditional media offerings: we don't have one product but over a 100 million different products with one for each of our members with personalized recommendations and personalized visuals," the company wrote in a December 2017 blog post.

Netflix has said that its recommendations drive 80% of discovery of shows on the platform.

Yet, algorithms may not always be enough. The streaming company is testing other, traditional methods of helping people find TV shows and movies to watch, including weekly lists that rank the 10 most-watched top titles in a region. The experiments come as Netflix releases more original series and movies.

The lists could help Netflix promote its new originals, even if the titles are not what the algorithms would otherwise recommend, experts say.

"There's always the trade-off with the motivations of the people making the recommendations," said Halverson, the Akamai exec. "If you're unlikely to watch this show but Netflix would like you to watch it, how do they incorporate their objectives into their pure AI, which might recommend 'The Office' over and over?"

The methodology behind the top 10 lists, which are being tested in the UK, favors new releases, which could include new Netflix originals.

Looking further ahead, Netflix is experimenting with more interactive content that could one day be used to improve content recommendations, such as the choose-your-own-adventure-style episode of "Black Mirror," "Bandersnatch," and the upcoming, interactive episode of "Unbreakable Kimmy Schmidt."

Experts say the data gleaned from viewers' participation in the interactive shows could help Netflix better understand why people are drawn to certain attributes in content, such as the emotional tonality of characters.

It could make Netflix's algorithms smarter, as well as help the media and tech giant develop more of the kinds of content it knows its viewers want to watch. Netflix already uses its viewer data on actors, characters, franchises, and genres to help determine how much to invest in new productions.

"'Bandersnatch' is very interesting but it's very, very early for that," Yves Berguist, a data scientist and the founder and CEO of Corto AI, a startup that builds real-world products from research projects, said. "It will happen though, because the industry is talking about it."

At a basic level, Netflix's algorithms suggest TV shows and movies to watch based on users' viewing history and feedback, such as whether they liked or disliked a title, as well as the behavior of viewers that watch similar content. Netflix has more than 2,000 constantly evolving "taste communities," as it calls them, which focus on common through-lines in TV shows and movies, such as strong female leads, supernatural elements, or extreme worlds.

"At Netflix, we strive to continually improve our recommendations so that when you visit the service, we have a suggestion you love," a spokesperson said. "These recommendations are powered by how members interact with our content - what kinds of shows they watch, how much of a series they watch, and much more. This information is used to construct individualized homepages for over 140 million members and serve our members even better."

Netflix spent more than $1.2 billion on technology and development in 2018, which included improvements to its platform and recommendation systems. It employs teams of researchers with varied backgrounds including neuroscience, biostatistics, economics, and physics, to improve its recommendations and tailor the artwork, videos, and other elements of the user experience that may affect whether someone presses play on a title or not.

How streaming services are using AI to make better emotional connections to viewers

The most compelling immediate opportunity for media companies that are using deep learning to make content recommendations is in using the technology to better understand why people are emotionally drawn to particular pieces of content - be it uplifting films or gritty TV dramas - at particular moments in time, executives at Disney Streaming Services, the streaming-technology division at the media giant that was formed from its acquisition of BAMTech, told Business Insider.

"Human beings are really awesome at - especially in person - picking up on visual cues and audio, graphic cues, and body language and all those sorts of things," Joe Inzerillo, chief technology officer of Disney Streaming Services, said. "We keep clawing our way closer and closer to understanding what makes up that phenomenon."

Algorithms are already drawing connections between the ways people interact with content on streaming-video platforms, and the attributes of the content itself, such as the story arcs, character journeys, and genres, to better understand our tastes.

Inzerillo hopes that Disney will be able to discover new dimensions of its content, over time from the way people interact on its streaming platforms, including ESPN Plus, Hulu, and the upcoming Disney Plus service.

He likened the opportunity to the way that tech companies have learned from how people interact with computers, starting from the days before mice, to today, where a gesture on an iPad or a voice command to a smart speaker, can help computers understand people in entirely new ways.

The Takeaway

Disney

Laura Evans.

"We can feel very emotional about a particular brand, or a movie, or a series, but really understanding what it is about that story arc [that resonates with people] is, in the world of entertainment, the most compelling application that I would see in the near future."

-Laura Evans, senior vice president of data at Disney Streaming Services