Modern post-hoc, model-agnostic XAI relies heavily on two mathematical frameworks: LIME and SHAP. Both are widely deployed in production pipelines to generate local explanations.
LIME (Local Interpretable Model-agnostic Explanations)
LIME operates on a simple intuitive principle: while the global decision boundary of a deep neural network may be incredibly complex and non-linear, the decision boundary around a single specific data point can be closely approximated by a simple linear model.
To explain a single prediction $x$, LIME performs the following steps:
- Takes the original instance $x$ and perturbs it to create a new dataset of slightly modified instances (e.g., masking out certain words in a text sample or altering numerical values slightly).
- Passes these perturbed instances through the black-box model to get their corresponding predictions.
- Weights the perturbed instances based on their proximity to the original point $x$ using a distance metric (closer points get higher weight).
- Trains an interpretable, surrogate model (like a simple linear regression) on this weighted, perturbed dataset.
The mathematical optimization is expressed as:
$$\xi(x) = \arg\min_{g \in G} \mathcal{L}(f, g, \pi_x) + \Omega(g)$$
Where:
- $f$ is the complex black-box model.
- $g$ is the simple, interpretable surrogate model (from the class of all possible linear models $G$).
- $\mathcal{L}$ measures the unfaithfulness of $g$ in approximating $f$ within the local neighborhood defined by proximity measure $\pi_x$.
- $\Omega(g)$ represents the complexity of the explanation model (e.g., limiting the linear model to only use the top 5 most impactful features so it remains human-readable).
SHAP (Shapley Additive exPlanations)
While LIME offers a fast local approximation, it lacks structural mathematical guarantees. SHAP solves this by leveraging cooperative game theory via Shapley values. In the context of XAI, the “game” is the model’s prediction task, and the “players” are the individual features of the input data point. The goal is to determine exactly how much payout (predictive change) should be credited to each player (feature).
To compute the true Shapley value for a single feature, the framework must evaluate the model’s output across every possible combination (coalition) of features with and without that specific feature included. The marginal contribution of feature $i$ to a coalition $S$ (where $S$ is a subset of all features $F$ excluding $i$) is computed. The Shapley value $\phi_i$ is the weighted average of these marginal contributions across all possible subsets:
$$\phi_i = \sum_{S \subseteq F \setminus \{i\}} \frac{|S|!(|F| – |S| – 1)!}{|F|!} \left[ f(S \cup \{i\}) – f(S) \right]$$
SHAP is highly valued in regulated industries because it uniquely satisfies four foundational axioms of fairness:
- Efficiency: The sum of the Shapley values of all features must equal the difference between the model’s local prediction and the expected baseline average prediction of the dataset.
- Symmetry: If two features contribute identically to all possible coalitions, their calculated importance values must be exactly equal.
- Dummy (Null Player): If a feature provides zero marginal contribution to any coalition, its Shapley value is strictly zero.
- Additivity: If a model’s prediction is the sum of two separate model functions, the overall Shapley value is the sum of the individual Shapley values.
Real-World Case Studies across Regulated Verticals
The practical execution of XAI is best understood through its deployment across distinct economic sectors where regulatory oversight forces transparency.
1. Algorithmic Credit Lending and the FCRA
Under the Fair Credit Reporting Act (FCRA) and the Equal Credit Opportunity Act (ECOA) in the United States, financial institutions are legally mandated to provide consumers with specific “adverse action notices” if they are denied credit. Statements like “Your loan application was denied by an automated algorithm” are legally insufficient. The institution must state the exact reasons for denial (e.g., “Debt-to-income ratio too high,” “Delinquent credit obligations”).
By embedding a SHAP framework into their risk pipelines, banks using gradient-boosted trees (such as XGBoost) can automatically compute local feature attributions for every automated denial. If a customer is rejected, the system extracts the three features with the most negative Shapley values for that specific run, instantly generating a legally compliant, human-auditable reason code.
2. Clinical Diagnostics and Medical Imaging
In digital healthcare, deep convolutional neural networks (CNNs) are deployed to detect malignancies in radiological images. However, a doctor cannot ethically or legally prescribe an invasive treatment plan based solely on a software output stating: “98% probability of melanoma.”
XAI tools such as Grad-CAM (Gradient-weighted Class Activation Mapping) solve this by utilizing the gradients flowing into the final convolutional layer of the CNN. By calculating the coarse 2D spatial localization map of the gradients, Grad-CAM generates heatmaps (saliency maps) that overlay onto the original medical image.
This allows the radiologist to verify that the AI is focusing its mathematical attention on the actual cellular boundaries of the lesion, rather than hallucinating a diagnosis based on an artifact in the background of the image or a marker pen annotation left by a technician.