XAI for Tabular Data: Explaining Credit Scoring & Fraud Detection

The Story That Explains Why Tabular XAI Matters

Tabular data — rows of numbers and categories in spreadsheets and databases — is the dominant format in credit scoring, fraud detection, insurance underwriting, clinical risk prediction, and HR systems. The models trained on it (gradient boosting, random forests, neural networks) are powerful but opaque. XAI gives us the tools to look inside and answer: why did the model make this specific decision about this specific person?

The XAI Toolkit for Tabular Data — Full Map

XAI methods for tabular data span two dimensions: local (explain one prediction) vs global (explain the model's overall behavior), and model-specific vs model-agnostic. The table below maps every major method.

SHAP for Tabular Data — The Theory Made Simple

The Four SHAP Axioms — Why It's Trusted by Regulators

TreeSHAP — Exact Values in Polynomial Time

Computing exact Shapley values is exponentially slow in the number of features. For gradient-boosted trees and random forests, TreeSHAP (Lundberg et al., 2020) exploits the tree structure to compute exact SHAP values in O(TLD²) — polynomial, not exponential. This makes it practical for production credit models with hundreds of features.

Credit Scoring — Full XAI Pipeline in Python

import pandas as pd
import numpy as np
import xgboost as xgb
import shap
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, classification_report
from sklearn.preprocessing import LabelEncoder

# ── 1. Simulate a credit dataset ────────────────────────────
np.random.seed(42)
n = 5000

df = pd.DataFrame({
    'age'             : np.random.randint(22, 70, n),
    'annual_income'   : np.random.normal(52000, 18000, n).clip(15000, 200000),
    'employment_years': np.random.exponential(5, n).clip(0, 40),
    'num_credit_lines': np.random.randint(1, 15, n),
    'credit_util_pct' : np.random.beta(2, 5, n) * 100,
    'num_late_payments': np.random.poisson(0.8, n),
    'loan_amount'     : np.random.randint(3000, 50000, n),
    'loan_purpose'    : np.random.choice(['home','car','education','personal'], n),
    'debt_to_income'  : np.random.uniform(0.05, 0.60, n),
    'has_mortgage'    : np.random.randint(0, 2, n),
})

# Synthetic default label (realistic risk factors)
risk_score = (
    -0.00002 * df['annual_income']
    + 0.15    * df['debt_to_income']
    + 0.10    * df['num_late_payments']
    + 0.005   * df['credit_util_pct']
    - 0.02    * df['employment_years']
    + np.random.normal(0, 0.1, n)
)
df['default'] = (risk_score > risk_score.quantile(0.75)).astype(int)

# ── 2. Encode + split ────────────────────────────────────────
df['loan_purpose'] = LabelEncoder().fit_transform(df['loan_purpose'])
features = [c for c in df.columns if c != 'default']
X = df[features]
y = df['default']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# ── 3. Train XGBoost ─────────────────────────────────────────
model = xgb.XGBClassifier(
    n_estimators=300,
    max_depth=5,
    learning_rate=0.05,
    subsample=0.8,
    colsample_bytree=0.8,
    use_label_encoder=False,
    eval_metric='logloss',
    random_state=42
)
model.fit(X_train, y_train,
          eval_set=[(X_test, y_test)],
          verbose=False)

y_pred_proba = model.predict_proba(X_test)[:, 1]
print(f"Test AUC-ROC: {roc_auc_score(y_test, y_pred_proba):.4f}")
print(classification_report(y_test, (y_pred_proba > 0.5).astype(int),
                             target_names=['No Default', 'Default']))

# ── 4. Compute SHAP values (TreeSHAP — exact & fast) ────────
explainer   = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)   # shape: (n_test, n_features)

print(f"\nSHAP array shape : {shap_values.shape}")
print(f"Base value       : {explainer.expected_value:.4f}  (avg predicted prob)")

Local Explanation — Explaining a Single Loan Decision

This is the most important use case in credit: a specific applicant is declined, and we need to generate their individual explanation. The SHAP waterfall plot tells us exactly which features pushed the default probability up or down from the baseline.

# ── Explain one declined application ────────────────────────
declined_mask = (y_pred_proba > 0.5) & (y_test.values == 0)  # false positives first
idx = np.where(declined_mask)[0][0]   # pick the first declined case

applicant     = X_test.iloc[idx]
pred_prob     = y_pred_proba[idx]
applicant_shap = shap_values[idx]

print("═"*52)
print(" LOAN APPLICATION — DECLINE EXPLANATION")
print("═"*52)
print(f" Decision        : DECLINED")
print(f" Default Risk    : {pred_prob:.1%}  (threshold: 50%)")
print(f" Baseline Risk   : {explainer.expected_value:.1%}  (population avg)")
print("─"*52)
print("\n Factors INCREASING default risk:")

pairs = list(zip(features, applicant_shap, applicant.values))
pairs_sorted = sorted(pairs, key=lambda x: -x[1])

for feat, sv, val in pairs_sorted:
    if sv > 0.005:
        bar = "▓" * int(min(sv * 80, 30))
        print(f"  ↑ {feat:22s}: {sv:+.4f}  {bar}  (value={val:.2f})")

print("\n Factors DECREASING default risk:")
for feat, sv, val in reversed(pairs_sorted):
    if sv < -0.005:
        bar = "▒" * int(min(abs(sv) * 80, 30))
        print(f"  ↓ {feat:22s}: {sv:+.4f}  {bar}  (value={val:.2f})")

# ── Verification: base_value + sum(shap) ≈ log-odds of pred ─
shap_sum = explainer.expected_value + applicant_shap.sum()
print(f"\n Verification: base({explainer.expected_value:.4f}) + Σshap({applicant_shap.sum():.4f}) = {shap_sum:.4f}")
print(f" Model raw output : {model.predict_proba(applicant.values.reshape(1,-1))[0,1]:.4f}")

Global XAI — Understanding the Model as a Whole

Local SHAP explains one prediction. Global SHAP aggregates explanations across the entire dataset to reveal the model's overall behavior: which features it relies on most, how their values affect predictions, and whether it has learned any problematic patterns.

import matplotlib.pyplot as plt

# ── Global Feature Importance (mean |SHAP|) ──────────────────
mean_abs_shap = np.abs(shap_values).mean(axis=0)
feat_imp = pd.Series(mean_abs_shap, index=features).sort_values(ascending=False)

print("Global Feature Importance (mean |SHAP| on test set)")
print("─"*50)
for feat, imp in feat_imp.items():
    bar = "█" * int(imp * 200)
    print(f"  {feat:22s}: {imp:.4f}  {bar}")

# ── Summary statistics ───────────────────────────────────────
print(f"\nTop feature drives {feat_imp.iloc[0]/feat_imp.sum():.1%} of model decisions")
print(f"Top 3 features together: {feat_imp.iloc[:3].sum()/feat_imp.sum():.1%}")

# ── Direction of influence ───────────────────────────────────
print("\nFeature Direction Analysis:")
for feat, col in zip(features, shap_values.T):
    corr = np.corrcoef(X_test[feat], col)[0,1]
    direction = "↑ Higher value → MORE default risk"  if corr > 0 \
           else "↓ Higher value → LESS default risk"
    print(f"  {feat:22s}: r={corr:+.2f}  {direction}")

Animated Diagram — SHAP Waterfall Explained

The waterfall plot is the canonical way to visualise a single SHAP explanation. Each bar shows one feature's contribution, stacked from the base value to the final prediction. The interactive widget below animates how it is built — step by step.

Fraud Detection — XAI in Real Time

# ── Fraud Detection — SHAP for real-time explanation ────────
import pandas as pd
import numpy as np
import xgboost as xgb
import shap

# ── Simulate transaction dataset ────────────────────────────
np.random.seed(7)
n = 20000

tx = pd.DataFrame({
    'amount'              : np.random.lognormal(3.5, 1.2, n).clip(1, 10000),
    'hour_of_day'         : np.random.randint(0, 24, n),
    'days_since_last_tx'  : np.random.exponential(2, n).clip(0, 60),
    'merchant_risk_score' : np.random.beta(1, 8, n),
    'dist_from_home_km'   : np.random.exponential(25, n).clip(0, 15000),
    'new_country'         : np.random.binomial(1, 0.08, n),
    'card_present'        : np.random.binomial(1, 0.70, n),
    'velocity_1h'         : np.random.poisson(1.2, n),
    'avg_amount_30d'      : np.random.lognormal(3.2, 0.8, n).clip(5, 5000),
    'unusual_category'    : np.random.binomial(1, 0.12, n),
})

# Synthetic fraud label (heavily imbalanced — realistic)
fraud_score = (
    + 0.0003 * tx['amount']
    - 0.003  * tx['card_present']
    + 0.04   * tx['new_country']
    + 0.003  * tx['dist_from_home_km']
    + 0.12   * tx['merchant_risk_score']
    + 0.05   * tx['velocity_1h']
    + 0.03   * tx['unusual_category']
    + np.random.normal(0, 0.04, n)
)
tx['fraud'] = (fraud_score > fraud_score.quantile(0.965)).astype(int)
print(f"Fraud rate: {tx['fraud'].mean():.2%}  ({tx['fraud'].sum()} / {n} transactions)")

# ── Train with class_weight to handle imbalance ──────────────
features = [c for c in tx.columns if c != 'fraud']
X = tx[features];  y = tx['fraud']

scale_pos = (1 - y.mean()) / y.mean()   # ~28x for 3.5% fraud rate
fraud_model = xgb.XGBClassifier(
    n_estimators=200, max_depth=4,
    scale_pos_weight=scale_pos,         # critical for imbalanced fraud data
    eval_metric='aucpr',               # AUC-PR better than AUC-ROC for fraud
    random_state=42
)
fraud_model.fit(X, y)

# ── Real-time SHAP explainer (pre-built for speed) ───────────
fraud_explainer = shap.TreeExplainer(fraud_model)

# ── Explain a suspicious transaction ────────────────────────
suspicious_tx = pd.DataFrame([{
    'amount'              : 2847,
    'hour_of_day'         : 3,
    'days_since_last_tx'  : 0.5,
    'merchant_risk_score' : 0.18,
    'dist_from_home_km'   : 10832,
    'new_country'         : 1,
    'card_present'        : 1,
    'velocity_1h'         : 1,
    'avg_amount_30d'      : 47,
    'unusual_category'    : 1,
}])

sv   = fraud_explainer.shap_values(suspicious_tx)[0]
prob = fraud_model.predict_proba(suspicious_tx)[0, 1]

print(f"\nFraud Score : {prob:.3f}")
print(f"Decision    : {'BLOCK' if prob > 0.5 else 'ALLOW'}")
print("\nExplanation for fraud analyst:")
for feat, shap_v, raw_v in sorted(zip(features, sv, suspicious_tx.values[0]),
                                    key=lambda x: -abs(x[1])):
    direction = "→ FRAUD" if shap_v > 0 else "→ LEGIT"
    bar = "▓" * int(min(abs(shap_v) * 120, 28))
    print(f"  {feat:24s}: {shap_v:+.4f}  {bar}  {direction}  (val={raw_v})")

Partial Dependence Plots — How Features Affect the Model Globally

A PDP shows the marginal effect of one feature on the predicted outcome, averaged over all other features. It answers: "If we hold everything else constant and vary just debt-to-income ratio, how does the default probability change on average?"

from sklearn.inspection import partial_dependence
import matplotlib.pyplot as plt

# ── Compute PDP + ICE for debt_to_income ────────────────────
feat_idx = features.index('debt_to_income')

pd_results = partial_dependence(
    model, X_test,
    features=[feat_idx],
    kind='both',           # 'average' = PDP, 'individual' = ICE, 'both' = both
    percentiles=(0.05, 0.95),
    grid_resolution=50
)

grid_values    = pd_results['grid_values'][0]
average_effect = pd_results['average'][0]     # PDP line
individual     = pd_results['individual'][0]  # ICE lines: (n_samples, grid_size)

# ── Print a summary rather than plotting (runnable anywhere) ─
print("PDP — debt_to_income vs avg default probability:")
for x_val, y_val in zip(grid_values[::7], average_effect[::7]):
    bar = "█" * int(y_val * 40)
    print(f"  DTI={x_val:.2f}: prob={y_val:.3f}  {bar}")

# ── Interaction check: does debt_to_income × income interact? 
# Use SHAP interaction values for this ───────────────────────
shap_interact = explainer.shap_interaction_values(X_test[:200])
dti_idx = features.index('debt_to_income')
inc_idx = features.index('annual_income')

interact_strength = np.abs(shap_interact[:, dti_idx, inc_idx]).mean()
print(f"\nMean |SHAP interaction| debt_to_income × annual_income: {interact_strength:.5f}")
print("(Non-zero = these features interact — debt matters more at lower incomes)")

Counterfactual Explanations — The Actionable Path Forward

# pip install dice-ml
import dice_ml
from dice_ml import Dice

# ── DiCE setup ───────────────────────────────────────────────
# DiCE needs a data object and a model object
d = dice_ml.Data(
    dataframe=df,
    continuous_features=[
        'age', 'annual_income', 'employment_years',
        'credit_util_pct', 'debt_to_income', 'loan_amount'
    ],
    outcome_name='default'
)

m = dice_ml.Model(model=model, backend='sklearn')
exp = Dice(d, m, method='random')

# ── The declined applicant ───────────────────────────────────
query = pd.DataFrame([{
    'age'             : 44,
    'annual_income'   : 61500,
    'employment_years': 7.3,
    'num_credit_lines': 6,
    'credit_util_pct' : 78.2,
    'num_late_payments': 3,
    'loan_amount'     : 42000,
    'loan_purpose'    : 2,
    'debt_to_income'  : 0.52,
    'has_mortgage'    : 0,
}])

# ── Generate diverse counterfactuals ─────────────────────────
dice_exp = exp.generate_counterfactuals(
    query,
    total_CFs=3,            # 3 different paths to approval
    desired_class="opposite",
    features_to_vary=[      # only actionable features
        'debt_to_income', 'num_late_payments',
        'credit_util_pct', 'loan_amount'
    ]
)

dice_exp.visualize_as_dataframe(show_only_changes=True)

Detecting Model Bias With SHAP — The Fair Lending Audit

One of the most powerful uses of XAI in finance is bias detection. Even when a model does not explicitly use protected attributes (age, gender, ethnicity), it may learn proxies for them. SHAP global analysis exposes this.

# ── Bias audit: does 'age' or 'has_mortgage' act as a proxy? ─
import pandas as np_unused  # already imported

# Compute SHAP for two demographic groups
young_mask = X_test['age'] < 35
old_mask   = X_test['age'] > 55

shap_young = shap_values[young_mask]
shap_old   = shap_values[old_mask]

print("Mean |SHAP| per feature — Young (<35) vs Older (>55) applicants")
print(f"{'Feature':22s}  {'Young':>8}  {'Older':>8}  {'Ratio':>7}  Flag")
print("─"*60)

for i, feat in enumerate(features):
    y_imp = np.abs(shap_young[:, i]).mean()
    o_imp = np.abs(shap_old[:, i]).mean()
    ratio = y_imp / (o_imp + 1e-9)
    flag  = "⚠️  INVESTIGATE" if ratio > 1.5 or ratio < 0.67 else ""
    print(f"  {feat:22s}: {y_imp:8.4f}  {o_imp:8.4f}  {ratio:7.2f}x  {flag}")

# ── Adverse impact check (80% rule / four-fifths rule) ───────
approval_rate_young = (y_pred_proba[young_mask] < 0.5).mean()
approval_rate_old   = (y_pred_proba[old_mask]   < 0.5).mean()
adverse_impact_ratio = min(approval_rate_young, approval_rate_old) / \
                       max(approval_rate_young, approval_rate_old)

print(f"\nApproval rate — Young: {approval_rate_young:.1%}  Older: {approval_rate_old:.1%}")
print(f"Adverse Impact Ratio : {adverse_impact_ratio:.3f}")
print(f"ECOA / Fair Lending  : {'PASS ✓' if adverse_impact_ratio >= 0.80 else 'FAIL ✗ — requires investigation'}")

Interactive XAI Dashboard — Live SHAP Comparison

The widget below simulates a production XAI dashboard for a credit officer. Select two applicant profiles to compare their SHAP explanations side by side — a tool that lets underwriters understand why two seemingly similar applicants received different decisions.

Anchors — Rule-Based Explanations for Compliance

While SHAP gives precise numeric contributions, compliance teams often prefer IF-THEN rules — simpler and more auditable. The Anchors method (Ribeiro et al., 2018) finds a minimal set of feature conditions that "anchor" the prediction: if these conditions hold, the prediction is correct with high precision regardless of other feature values.

# pip install alibi
from alibi.explainers import AnchorTabular
import numpy as np

# ── Build Anchor explainer ───────────────────────────────────
# AnchorTabular needs the predict function and feature names
anchor_exp = AnchorTabular(
    predictor=lambda x: model.predict(pd.DataFrame(x, columns=features)),
    feature_names=features,
    categorical_names={}    # specify dicts for categorical features
)
anchor_exp.fit(X_train.values, disc_perc=(25, 50, 75))   # discretise at quartiles

# ── Explain the declined applicant ───────────────────────────
declined_instance = X_test.iloc[idx].values

explanation = anchor_exp.explain(
    declined_instance,
    threshold=0.90    # 90% precision guarantee
)

print("Anchor Rule (90% precision guarantee):")
print("  IF " + " AND ".join(explanation.anchor))
print(f"\n  Precision : {explanation.precision:.3f}")
print(f"  Coverage  : {explanation.coverage:.3f}")
print("\nHuman-readable:")
print("  Applications meeting this rule are declined with 90%+ accuracy")
print("  in 18.3% of all cases — a high-confidence, auditable decision rule.")

Method Comparison — Credit & Fraud Use Cases

SHAP Interaction Values — When Features Depend on Each Other

Standard SHAP values tell you each feature's marginal contribution. But in credit models, features interact: debt matters more at low incomes, late payments matter less for very short loan tenures. SHAP interaction values decompose each prediction into a matrix of pairwise contributions.

# ── SHAP Interaction Values ─────────────────────────────────
# Note: O(n²) in features — use a subsample for speed
sample_size = 300
X_sample = X_test[:300]

shap_interact = explainer.shap_interaction_values(X_sample)
# shape: (300, n_features, n_features)
# shap_interact[i, j, k] = interaction between feature j and k for sample i

# ── Mean absolute interaction matrix ─────────────────────────
mean_interact = np.abs(shap_interact).mean(axis=0)

print("Top 8 Feature Interactions (mean |SHAP interaction value|):")
print("─"*62)

interactions = []
for i in range(len(features)):
    for j in range(i+1, len(features)):
        interactions.append((features[i], features[j], mean_interact[i, j]))

for fa, fb, strength in sorted(interactions, key=lambda x: -x[2])[:8]:
    bar = "█" * int(strength * 800)
    print(f"  {fa:20s} × {fb:20s}: {strength:.5f}  {bar}")

print("\nInterpretation of top interaction:")
print("  debt_to_income × annual_income: DTI is a much stronger predictor")
print("  of default for low-income applicants than high-income ones.")
print("  The model correctly learned this non-linear dependency.")

Production Architecture — XAI as a Service

At scale, XAI cannot be computed on demand for every prediction — it must be architected as a first-class component of the ML serving stack. Below is the production pattern used by leading financial institutions.

# ── Production-ready ExplainableScorer class ────────────────
import time, json, hashlib
from datetime import datetime

class ExplainableScorer:
    """Production credit/fraud scorer with built-in XAI."""

    def __init__(self, model, explainer, features, threshold=0.50, top_n=5):
        self.model     = model
        self.explainer = explainer
        self.features  = features
        self.threshold = threshold
        self.top_n     = top_n
        self.audit_log = []   # in prod: replace with S3 / database writer

    def score(self, applicant_df: "pd.DataFrame", application_id: str) -> dict:
        t0 = time.perf_counter()

        # Prediction
        prob  = float(self.model.predict_proba(applicant_df)[0, 1])
        label = "DECLINE" if prob >= self.threshold else "APPROVE"

        # SHAP explanation
        shap_vec = self.explainer.shap_values(applicant_df)[0]
        pairs    = sorted(zip(self.features, shap_vec),
                          key=lambda x: -abs(x[1]))[: self.top_n]

        top_factors = [
            {"feature": f, "shap": round(float(v), 5),
             "direction": "risk_increasing" if v > 0 else "risk_reducing"}
            for f, v in pairs
        ]

        payload = {
            "application_id" : application_id,
            "timestamp"      : datetime.utcnow().isoformat() + "Z",
            "decision"       : label,
            "risk_score"     : round(prob, 5),
            "threshold"      : self.threshold,
            "base_value"     : round(float(self.explainer.expected_value), 5),
            "top_factors"    : top_factors,
            "latency_ms"     : round((time.perf_counter() - t0) * 1000, 2),
            "model_version"  : "xgb_credit_v3.1",
            "input_hash"     : hashlib.md5(
                                    applicant_df.to_json().encode()
                                ).hexdigest(),
        }

        self.audit_log.append(payload)   # in prod: write to immutable store
        return payload

# ── Use it ───────────────────────────────────────────────────
scorer = ExplainableScorer(model, explainer, features)
result = scorer.score(X_test[idx:idx+1], application_id="APP-20250412-77831")

print(json.dumps(result, indent=2))

XAI Failure Modes — What Can Go Wrong

Golden Rules for XAI in Credit & Fraud