Data scientist at TED: Algorithms are a 'weapon of math destruction' that need to be regulated
Herwig Prammer/Reuters
Algorithms are increasingly prevalent in everything we do, from getting insurance to applying to jobs (if the algorithm doesn't pick your application from a big pile, good luck). While it may seem like these computer calculations are objective measures of whether someone is qualified for a certain position or home loan, human bias is always behind them.
Cathy O'Neil, a data scientist and founder of ORCAA Algorithmic Auditing, had a message for the audience at the 2017 TED talks: "We need to demand accountability from our algorithmic overlords."
During one of the most popular talks of the Wednesday evening TED sessions, O'Neil laid out the various ways that algorithms can go wrong.
She told the story of the value-added algorithmic model for evaluating New York City teachers, which has penalized seemingly excellent educators, as the New York Times found out. And O'Neil brought up the ProPublica investigations into algorithmic racial discrimination, in which the news organization found algorithms discriminating in prison sentencing, car insurance premiums, and elsewhere.
O'Neil also posited a thought experiment: What if Fox News decided to get a fresh start after its recent sexual harassment scandals by using an algorithm in hiring? And what if that algorithm relied on the last 20 years of hiring data to figure out who might be successful in the future? "It would filter out women because they don't look like people who were successful in the past," she speculated.
"Algorithms don't make things fair. They automate the status quo," she said. "They're weapons of math destruction."
The free market won't solve the problem, argued O'Neil, because "there's a lot of money to be made in unfairness."
Her solution: algorithm data auditing. That would involve scrutinizing algorithms to check for integrity (by dealing with biases in the data used to build the algorithms), examining definitions of success (to make sure the algorithm supports non-biased goals), checking overall accuracy, monitoring long-term effects, and avoiding feedback loops.
Ultimately, O'Neil believes we need a government regulator in charge of algorithms. "I'm not holding my breath in the Trump administration," she said.
- A centenarian who starts her day with gentle exercise and loves walks shares 5 longevity tips, including staying single
- A couple accidentally shipped their cat in an Amazon return package. It arrived safely 6 days later, hundreds of miles away.
- FSSAI in process of collecting pan-India samples of Nestle's Cerelac baby cereals: CEO
- 7 Nutritious and flavourful tiffin ideas to pack for school
- India's e-commerce market set to skyrocket as the country's digital economy surges to USD 1 Trillion by 2030
- Top 5 places to visit near Rishikesh
- Indian economy remains in bright spot: Ministry of Finance
- A surprise visit: Tesla CEO Elon Musk heads to China after deferring India visit