The Dark Secret at the Heart of AI - MIT Technology Review

there is no obvious way to design such a system so that it could always explain why it did what it did... How well can we expect to communicate—and get along with—intelligent machines that could be unpredictable and inscrutable? ... a journey to the bleeding edge of research on AI algorithms... including a meeting with one of the great philosophers of our time....
the system extracts and highlights snippets of text that are representative of a pattern it has discovered... a loop where the machine and the human collaborate... the nature of these machine-learning systems ... produce a lot of false alarms... system designed to classify an e-mail message as coming from a terrorist... could highlight certain keywords ...
we may have to simply trust AI’s judgment or do without ... that judgment will have to incorporate social intelligence... fit with our social norms... consistent with our ethical judgments... we should be as cautious of AI explanations as we are of each other’s