In our diabetes readmission example, we realized that our model was failing to generalize to new settings because it learned the wrong features. This is an instance of data leakage, whereby information that was available in the context of model training was unavailable in the deployment setting.
Determining whether leakage is occurring is cognitively expensive; the researcher must extrapolate whether a given feature will be available in other settings. When models consume hundreds of features, it's impractical for a data scientist or researcher to consider each and every one. This prevents us from preconceiving points of failure in a deployment setting. But with algorithms like feature importance, which describes the magnitude of dependence a model has on a feature, the data scientist can focus his or her attention on the most important features and better forecast model behavior.
In production settings where populations are dynamic (for instance, e-commerce marketplaces), we need to make inferences about model decisions for observations that are different than those observed during training. In other words, does a model extrapolate reasonably to new data?
Sometimes, models yield extreme values for new regions in the domain, as is often the case with real-valued parametric models and neural networks. Meanwhile, other algorithms like random forests can be completely flat (kernel-based models often behave more predictably in new regions). While the production-extrapolation problem is better solved with ongoing model evaluation and online learning algorithms, interpretation algorithms that identify important features can help us spot extrapolation issues before they occur.
Related to this extrapolation challenge is the problem of leverage points. A model's decision criteria can be adversely affected by observations with extreme target values. In particular, high variance models that are not robust to outliers, like a neural network without sufficient regularization and some random forest implementations, can be sensitive to leverage points and learn to predict extreme values in certain settings. Interpretation algorithms that allow us to visualize predictions across regions of the input space can help us identify extreme behavior.
So that they can forecast model behavior in production settings, data scientists often use simpler, more inherently interpretable models in the place of potentially more accurate models that can represent complex hypotheses. But we don't have to continue making these trade-offs: It's possible to develop a toolbox that shows the mechanics of any model and identifies boundaries between training data and regions of extrapolation.