Foundations of Explainable AI: What It Is, Why It Matters, and How It Works

What Is Explainable AI — and Why Does It Matter?

Explainable AI (XAI) is the collection of methods, tools, processes, and design principles that make the predictions and behaviours of artificial intelligence systems understandable to humans — not just to the engineers who built them, but to the doctors, judges, loan officers, and everyday users who depend on them.

XAI asks a deceptively simple question: "Why did the model output this specific result?" The difficulty — and the entire research field — flows from the fact that the answer is often buried inside millions of learned parameters with no obvious translation to human language.

Why Does It Matter Right Now?

XAI sits at the intersection of four urgent pressures that have converged in the 2020s:

The "Black Box" Problem in Machine Learning

A black box model is any machine learning model whose internal reasoning is not accessible or interpretable to humans. Inputs go in. Outputs come out. What happens in between is opaque — a wall of matrix multiplications, activation functions, and learned weights that produce a number with no human-readable explanation.

Black Box vs Glass Box — A Spectrum

Not all models are equally opaque. The field defines a spectrum from fully interpretable glass box models to deeply opaque black box models.

Interpretability vs Explainability vs Transparency

These three terms are often used interchangeably in casual speech but mean distinct things in the field of responsible AI. Getting the distinction right is foundational for choosing the correct XAI method for any given situation.

Formal Definitions at a Glance

Types of Explanations: Global vs Local

Global Explanations — Understanding the Model Overall

A global explanation characterises the model's overall behaviour across the entire dataset or input space. It answers: "What has this model generally learned to do?"

Local Explanations — Understanding One Specific Prediction

A local explanation explains why the model made a specific prediction for a specific individual. It answers: "Why did this particular applicant get rejected?"

Comparison Table: Global vs Local

Python Code: SHAP Global & Local Explanations

import shap
import numpy as np
import pandas as pd
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split

# ── 1. Train a black-box model ──────────────────────────────
X, y = shap.datasets.adult()         # Income prediction dataset
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

model = GradientBoostingClassifier(n_estimators=200, max_depth=4, random_state=42)
model.fit(X_tr, y_tr)

# ── 2. Create a SHAP explainer ──────────────────────────────
# TreeExplainer is exact (no approximation) for tree-based models
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_te)

# ── 3. GLOBAL explanation — feature importance summary ───────
print("=== GLOBAL — Mean |SHAP| across all test predictions ===")
mean_abs_shap = np.abs(shap_values).mean(axis=0)
importance_df = pd.DataFrame({
    'feature': X_te.columns,
    'mean_abs_shap': mean_abs_shap
}).sort_values('mean_abs_shap', ascending=False)
print(importance_df.to_string(index=False))

# ── 4. LOCAL explanation — one specific individual ───────────
idx = 42                             # Explain prediction for row 42
individual = X_te.iloc[[idx]]
pred_prob = model.predict_proba(individual)[0][1]

print(f"\n=== LOCAL — Individual #{idx} ===")
print(f"Predicted probability of income >50k: {pred_prob:.3f}")
print(f"Base value (expected):               {explainer.expected_value:.3f}")
print("\nTop SHAP contributions (this person):")

local_shap = pd.Series(shap_values[idx], index=X_te.columns).sort_values(key=lambda x: -x.abs())
for feat, val in local_shap.head(5).items():
    direction = "▲ pushes APPROVE" if val > 0 else "▼ pushes REJECT"
    print(f"  {feat:25s}: {val:+.4f}  {direction}")

Python Code: LIME — Approximating Any Black Box Locally

import lime
import lime.lime_tabular
import numpy as np

# ── LIME explainer setup ────────────────────────────────────
# LIME needs to know the training data statistics to perturb features
lime_explainer = lime.lime_tabular.LimeTabularExplainer(
    training_data=X_tr.values,
    feature_names=X_tr.columns.tolist(),
    class_names=['≤50k', '>50k'],
    mode='classification',
    discretize_continuous=True,
    random_state=42
)

# ── Explain one prediction with LIME ─────────────────────────
exp = lime_explainer.explain_instance(
    data_row=X_te.values[idx],
    predict_fn=model.predict_proba,
    num_features=6,
    num_samples=2000           # More samples = better approximation
)

print("=== LIME Local Explanation ===")
for feature_rule, contribution in exp.as_list():
    sign = "▲" if contribution > 0 else "▼"
    print(f"  {sign} {feature_rule:40s}: {contribution:+.4f}")

Human-Centered AI and Trust

Human-Centered AI (HCAI) is the design philosophy that places human needs, capabilities, and values at the centre of every AI system. It goes beyond technical explainability to ask: "Is this explanation useful to the actual human making this decision?"

The Four Dimensions of Human Trust in AI

Building Trust — The XAI Pipeline

Major XAI Methods — Quick Reference

Python Code: Checking Explanation Faithfulness

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
import shap

# ── Strategy: test if SHAP explanations are faithfully ───────
# ── reflecting the model by running a prediction rebuild test

# 1. Get SHAP values and expected value (base rate)
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_te)
base_value  = explainer.expected_value

# 2. Reconstruct predictions from SHAP values alone
# SHAP guarantee: base_value + sum(shap_i) = model log-odds output
shap_sum = shap_values.sum(axis=1) + base_value

# 3. Compare to actual model output (log-odds = logit of probability)
from scipy.special import expit  # sigmoid
shap_probs  = expit(shap_sum)
model_probs = model.predict_proba(X_te)[:, 1]

# Maximum absolute deviation between SHAP reconstruction and true model
max_err = np.abs(shap_probs - model_probs).max()
mean_err = np.abs(shap_probs - model_probs).mean()

print(f"SHAP faithfulness check:")
print(f"  Max absolute error:  {max_err:.6f}")
print(f"  Mean absolute error: {mean_err:.6f}")
print(f"  TreeExplainer is EXACT for tree models — error should be ~0")

Non-Negotiable Rules for Responsible XAI Deployment