When Not to Trust the Algorithm

I was particularly ashamed of the way that the AAA ratings, which were in some sense a mathematical promise of safety, had been actually just lies, mathematical lies... like a weaponized mathematics ... actually people don’t want to know what their actual risk ... mathematics ... was being used ... so that people could go on doing essentially corrupt things but claim that they had some kind of mathematical stamp of approval... relatively invisible ... we’d all drunk the Kool-Aid of big data ... it was blinding us to the real problem...
I defined weapons of math destruction to be a certain class of algorithms that I think are deeply troubling. And they’re characterized with three characteristics...
they’re really important.... whether or not you’re going to get a job... how long you go to jail ...
The second is that it’s ... silent, secret scoring systems... people... don’t know the formula... often don’t even know that they’re being scored... it’s not accountable ...
if it’s ... destroying individual lives, often unfairly ... engendering ... destructive positive feedback loops... making things worse and worse....
people more and more have been replacing their HR with algorithms... these algorithms have not been audited for fairness or legality...
we’re basically training machine learning algorithms on old application data and we’re training them to a definition of success... we know now that women were systematically prevented from succeeding at Fox News... a machine learning algorithm using that old data... would filter out the women... no standards of safety set in place ... it’s un-trackable... if you filter people out from even getting an interview, you never see them again...
I’m hoping that every data science institute... take ethics seriously and have a class on ethics.... The regulations that already exist around anti-discrimination law, disparate impact, and fair hiring practices have to be enforced in the realm of big data.