A former Google data scientist explains why Netflix knows you better than you know yourself

Much of "Everybody Lies" - a new book by economist Seth Stephens-Davidowitz - explores how Big Data reveals people's secrets. People share with Google their desire to read racist jokes and watch same-sex porn when they'd never ever share those desires IRL.

But one of the most fascinating parts of the book shows how Big Data can also reveal when people are lying to themselves.

Stephens-Davidowitz, who is a former Google data scientist, points to the evolution of Netflix's algorithm to illustrate his point.

In its early days, the company allowed users to create a queue of movies they wanted to watch in the future. But when users were reminded a few days later of the movies they'd picked out, they rarely clicked on them.

Stephens-Davidowitz writes:

"What was the problem? Ask users what movies they plan to watch in a few days, and they will fill the queue with aspirational, highbrow films, such as black-and-white World War II documentaries or serious foreign films. A few days later, however, they will want to watch the same movies the usually want to watch: lowbrow comedies or romance films. People were consistently lying to themselves."

As a result, Stephens-Davidowitz writes, Netflix stopped asking people to choose what they wanted to watch in the future. Instead, Netflix made recommendations based on movies that similar users had watched.

"The result: customers visited Netflix more frequently and watched more movies," he adds.

As Xavier Amatriain, a former Netflix data scientist, told Stephens-Davidowitz, "The algorithms know you better than you know yourself."

That's kind of creepy - and creepier when you realize that Netflix is hardly the only business making bank on people's lack of self-knowledge.

When Stephens-Davidowitz visited the Business Insider office in May, he said that many gyms have learned to harness the power of people's overoptimism. Specifically, he said, "they've figured out you can get people to buy monthly passes or annual passes, even though they're not going to use the gym nearly enough to warrant this purchase."

Most people, he said, would be better off paying per visit to the gym, because they may only go a handful of times.

"But the overwhelming majority of people buy these monthly and yearly passes, thinking they're going to go 30 or 60 times. Gyms have made a lot of money capitalizing on the consumer's overoptimism."