Motivating Interpretability - A Researcher and Engineering Perspective
Published:
Deep neural networks (DNNs) are known to be almost fully opaque compared to traditional algorithms, even to top engineers and scientists who use DNNs. Many reasons have been proposed for why understanding the decision making process of these models is important, such as human curiosity, scientific discovery, bias detection, or algorithm safety measures and auditing. Furthermore, a model which is interpretable may exhibit more fairness, reliability, and trust to the general public 1. Several methods to interpret DNNs in computer vision have been proposed to varying degrees of success. For example, many recent SOTA saliency map visualization methods have been shown to be on par with some random baselines. Similar issues with other interpretability methods have been criticized on social media, pointing out that the reliance on these methods in mission critical situations will likely do more harm than good.