Spotify made its shuffle feature less random so that it would actually feel more random to listeners - here's why
- Ben Cohen is a sports reporter for the Wall Street Journal; he writes about the NBA, the Olympics, and other topics that don't involve extraordinarily athletic people.
- The following is an excerpt from his new book, "THE HOT HAND: The Mystery and Science of Streaks."
- In it, he describes the puzzling complaint that Spotify used to receive - people said its shuffle feature wasn't shuffling their music.
- In fact, it turns out that we're terrible at understanding when things are truly random, like when an iPod would play the same song twice in a row.
- To fix the problem, Lukáš Poláček rewrote the algorithm slightly so that different songs by the same artist would be evenly distributed throughout playlists.
- Visit Business Insider's homepage for more stories.
Spotify had a problem. One of the world's most popular streaming music services kept hearing one puzzling complaint from its users. They believed the shuffle button was broken. It wasn't actually shuffling their music.
But the problem wasn't unique to Spotify. In fact, a few years earlier, a competing business had found itself struggling with exactly the same issue. The original iPod was another glorious device that gave people the ability to carry portable jukeboxes in their pockets. But not everyone who owned this magical Apple machine was pleased. Many of them suspected they had received defective iPods. When they heard the same artist twice in a row, they came to a judgment that revealed more about themselves than the product they were holding. They concluded that their randomly generated music simply couldn't be random.
"It really is random," Steve Jobs said during a 2005 keynote. "But sometimes random means you've got two songs from the same artist next to each other."
The real problem was one that no amount of money or engineering talent could solve. Why did we believe that something was wrong with Spotify and Apple? It's because we are terrible at understanding randomness.
There is something about the way that randomness paralyzes the human mind that was beyond the control of Spotify, Apple or any other company worth billions of dollars. Which makes it precisely the sort of thing that psychologists study. One of the most beautiful examples of this phenomenon is a classic psychological bias that has been studied for several decades by some of the smartest people on earth.
It's known as "the hot hand."
The hot hand has been a topic of fierce academic debate since the 1985 publication of a deeply counterintuitive paper with a delicious conclusion: that there is no such thing as the hot hand. Their study examined streaks in basketball, and when a team of authors that included the great Amos Tversky looked at the hot hand, they couldn't believe the data that was staring them back in the face. As much as basketball players and basketball fans believed they were more likely to make their next shot after making a few consecutive shots - that they couldn't miss because they were in the zone - the evidence actually suggested the opposite. This made for a bombshell of a scholarly paper.
There were three major findings in their study. The first was that the hot hand in basketball was a fallacy. The second was that humans fundamentally overestimate streakiness. And the third was the one that Apple and Spotify understood. It was that we have a nasty habit of systematically misperceiving randomness - and such biases can lead us in all kinds of funky directions.
"People see patterns where there are none," Tversky once said, "and they invent causes to explain them."
That brings us back to Spotify, Apple, and their peculiar dilemma.
All the way back in 2014, which is basically a century ago in tech startup time, Lukáš Poláček wasn't technically a full-time Spotify employee. He was still a student in Stockholm studying theoretical computer science. But he happened to be working on randomness algorithms when he noticed an internal discussion of the shuffle issue. Here was a way to make his expertise in computer science a whole lot less theoretical. Poláček volunteered to help. A strange number of colleagues were confused when he told them he was working on the shuffle button. "What is there to work on?" they would say. "It's just random!"
In that sense, they were right. There wasn't much to work on. It was just random. Poláček required one day of work and roughly fifteen lines of code to rewrite the algorithm. The man who would be called "Mr. Shuffle" at Spotify parties simply took different songs by the same artist and distributed them more or less evenly across the playlist. If your family had a playlist of Beyoncé, The Beatles, and Billy Joel, for example, you would never have the burden of hearing three consecutive songs by Billy Joel.
Apple's solution to this funny little problem turned out to be a similar one.
The reason that no less than Steve Jobs was onstage that day talking about randomness was that Apple was introducing a fresh new feature called "Smart Shuffle" that gave people what they wanted. "What we've added is Smart Shuffle to actually make it less random - if you want," he said. "Smart Shuffle allows you to control how likely you are to hear multiple songs by the same artist or from the same album in a row."
It might not have made sense in theory. But human beings live in reality. As he heard himself explaining how it worked, Steve Jobs couldn't help but laugh at the absurdity of what he was saying.
"Even though people will think it's more random," he said, "it's actually less random."
Adapted from the book THE HOT HAND: The Mystery and Science of Streaks by Ben Cohen. Copyright © 2020 by Ben Cohen. From Custom House, a line of books from William Morrow/HarperCollins Publishers. Reprinted by permission.