Explainability

NIST splits one word into three, and the split is the useful part. Transparency, explainability, and interpretability are distinct characteristics that support each other. Transparency answers what happened in the system. Explainability answers how a decision was made. Interpretability answers why it was made, and what it means to the person affected. Most arguments about whether an AI system is explainable are really arguments about which of those three somebody wants.

The framework is direct about what the property is for. Explainable and interpretable AI systems offer information that will help end users understand the purposes and potential impact of an AI system. Risk from lack of explainability may be managed by describing how AI systems function, with descriptions tailored to the user's role, knowledge, and skill level. There is a practical benefit as well. Explainable systems can be debugged and monitored more easily, and they lend themselves to more thorough documentation, audit, and governance.

Vendor tooling makes the split concrete in a way the framework does not. Microsoft describes interpretability as generating human-understandable descriptions of the predictions of a machine learning model, then divides the output in two. Global explanations answer, for example, what features affect the overall behavior of a loan allocation model. Local explanations answer why one particular customer's application was approved or rejected. Feature importance tells you how each data field affects the model's predictions, which is what makes either answer possible.

The same page names a second reason to care. You use interpretability to uncover potential sources of unfairness by understanding whether the model is basing predictions on sensitive features, or on features that are highly correlated with them. It also names what you are up against. Opaque-box models are those models for which you have no information about their internal workings.