Linear & Logistic Regression Coefficients in Explainable AI (XAI)

The Story — Why Your Model Needs to Explain Itself

Explainable AI (XAI) is the field of making machine learning models interpretable — not just accurate. A model that predicts correctly but cannot justify its reasoning is a liability in medicine, finance, law, and anywhere decisions affect human lives. Regression coefficients, when properly understood, are your first and most transparent window into a model's logic.

Linear Regression — The Equation Behind the Explanation

Before we interpret coefficients, we must understand what they are. A linear regression model predicts a continuous outcome (house price, sales revenue, temperature) as a weighted sum of input features.

The Three Golden Rules of Coefficient Interpretation

Coefficients are powerful but subtle. Many practitioners misread them. These three rules prevent the most common errors.

Python — Fitting Linear Regression and Extracting Coefficients

Let's build a real example using the California Housing dataset. We will fit a linear regression model, extract coefficients, scale them for fair comparison, and visualise them as an XAI explanation.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score, mean_absolute_error

# ── 1. Load data ──────────────────────────────────────────
data = fetch_california_housing(as_frame=True)
X, y = data.frame.drop("MedHouseVal", axis=1), data["target"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# ── 2. Standardise features ───────────────────────────────
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s  = scaler.transform(X_test)

# ── 3. Fit model ───────────────────────────────────────────
model = LinearRegression()
model.fit(X_train_s, y_train)

# ── 4. Extract and rank coefficients ─────────────────────
coef_df = pd.DataFrame({
    "Feature": X.columns,
    "Coefficient": model.coef_
}).sort_values("Coefficient", key=lambda x: x.abs(), ascending=False)

print("Standardised Coefficients (XAI Feature Importance):")
print(coef_df.to_string(index=False))
print(f"\nR² Score:  {r2_score(y_test, model.predict(X_test_s)):.4f}")
print(f"MAE:       {mean_absolute_error(y_test, model.predict(X_test_s)):.4f}")

# ── 5. XAI Plot ────────────────────────────────────────────
colors = ['#34d399' if c > 0 else '#f87171' for c in coef_df["Coefficient"]]
fig, ax = plt.subplots(figsize=(9, 5))
ax.barh(coef_df["Feature"], coef_df["Coefficient"], color=colors, edgecolor='none')
ax.axvline(0, color='white', lw=1)
ax.set_title("Linear Regression — Standardised Coefficients (XAI)", pad=12)
ax.set_xlabel("Coefficient Value (std units)")
plt.tight_layout()
plt.show()

Logistic Regression — When the Outcome is a Decision

Linear regression predicts a number. But what if you need to predict a category — spam or not spam, fraud or legitimate, survived or did not survive? This is where logistic regression becomes the tool of choice.

Odds Ratios — The Language of Logistic Regression XAI

In logistic regression, the raw coefficient βᵢ tells you how a one-unit change in feature xᵢ shifts the log-odds of the positive outcome. But log-odds are hard to intuit. The solution: exponentiate. e^β gives you the Odds Ratio (OR).

Python — Logistic Regression with Odds Ratios and XAI Waterfall

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, roc_auc_score

# ── Load ──────────────────────────────────────────────────
data = load_breast_cancer(as_frame=True)
X, y = data.data, data.target  # 1=malignant, 0=benign

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

# ── Scale ─────────────────────────────────────────────────
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s  = scaler.transform(X_test)

# ── Fit ───────────────────────────────────────────────────
lr = LogisticRegression(max_iter=1000, random_state=42)
lr.fit(X_train_s, y_train)

# ── Build XAI coefficient table ───────────────────────────
coef_df = pd.DataFrame({
    "Feature":     X.columns,
    "Coefficient": lr.coef_[0],
    "OddsRatio":   np.exp(lr.coef_[0])
}).sort_values("Coefficient", key=lambda x: x.abs(), ascending=False)

print("Top 10 Features by Absolute Coefficient:")
print(coef_df.head(10)[["Feature","Coefficient","OddsRatio"]].to_string(index=False))

# ── Evaluate ──────────────────────────────────────────────
y_pred  = lr.predict(X_test_s)
y_proba = lr.predict_proba(X_test_s)[:,1]
print(f"\nROC-AUC: {roc_auc_score(y_test, y_proba):.4f}")
print(classification_report(y_test, y_pred, target_names=['Benign','Malignant']))

# ── Explain a single prediction — XAI waterfall ──────────
sample = X_test_s[0]         # one patient
contributions = lr.coef_[0] * sample  # feature × coefficient
top_n = 8

contrib_df = pd.DataFrame({
    "Feature": X.columns,
    "Contribution": contributions
}).sort_values("Contribution", ascending=False)

print(f"\nSingle-prediction explanation (patient 0):")
print(f"Base (intercept): {lr.intercept_[0]:.4f}")
print(contrib_df.head(top_n).to_string(index=False))
print(f"Final log-odds:   {lr.intercept_[0] + contributions.sum():.4f}")
print(f"Predicted prob:   {y_proba[0]:.4f}")

The Coefficient Waterfall — Explaining One Prediction at a Time

The most powerful XAI technique for regression models is the waterfall chart. Each bar represents how much one feature's coefficient contribution pushes the prediction up (positive) or down (negative) from the model's base (intercept). This is the local explanation that bridges the gap between a global model and a single individual's outcome.

Statistical Significance — Are Your Coefficients Real?

A coefficient of 3.2 sounds meaningful. But is it statistically different from zero? Could it be noise? Statistical significance tests and confidence intervals tell you whether to trust a coefficient for XAI purposes.

import statsmodels.api as sm
from statsmodels.stats.outliers_influence import variance_inflation_factor
import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.preprocessing import StandardScaler

# Load and scale (use first 500 rows for speed)
data  = fetch_california_housing(as_frame=True)
X     = data.frame.drop("MedHouseVal", axis=1).iloc[:500]
y     = data["target"].iloc[:500]
scaler= StandardScaler()
Xs    = scaler.fit_transform(X)

# ── OLS with p-values and confidence intervals ────────────
Xs_c  = sm.add_constant(Xs)  # adds intercept column
model = sm.OLS(y, Xs_c).fit()
print(model.summary().tables[1])  # coefficients table with p-values

# ── VIF check ─────────────────────────────────────────────
vif_df = pd.DataFrame({
    "Feature": X.columns,
    "VIF"    : [variance_inflation_factor(Xs, i) for i in range(Xs.shape[1])]
}).sort_values("VIF", ascending=False)

print("\nVIF Scores (should be < 5 for reliable XAI):")
print(vif_df.to_string(index=False))

Regularisation — When Coefficients Need a Leash

Regularisation adds a penalty to the model for having large coefficients. This prevents overfitting but also changes the coefficients' XAI meaning. There are two main types:

Coefficients vs SHAP — The XAI Ladder

Linear regression coefficients are the first rung of the XAI ladder. As models grow more complex (decision trees, random forests, neural networks), coefficients no longer exist in the same form. Here is how the XAI toolkit evolves:

Python — SHAP Values from a Logistic Regression

SHAP's LinearExplainer computes exact SHAP values for linear models. These are numerically equivalent to the coefficient-contribution waterfall we built manually — but computed automatically for any number of features, and with proper handling of feature correlations.

import shap
import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

# ── Setup (same as Section 07) ────────────────────────────
data     = load_breast_cancer(as_frame=True)
X, y     = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
scaler   = StandardScaler()
Xtr_s    = scaler.fit_transform(X_train)
Xte_s    = scaler.transform(X_test)
Xtr_df   = pd.DataFrame(Xtr_s, columns=X.columns)
Xte_df   = pd.DataFrame(Xte_s, columns=X.columns)

lr = LogisticRegression(max_iter=1000, random_state=42)
lr.fit(Xtr_s, y_train)

# ── SHAP LinearExplainer ──────────────────────────────────
explainer   = shap.LinearExplainer(lr, Xtr_df, feature_perturbation="interventional")
shap_values = explainer.shap_values(Xte_df)

# Global summary: feature importance across all test samples
shap.summary_plot(shap_values, Xte_df, plot_type="bar", max_display=10)

# Local explanation: waterfall for one patient
shap.waterfall_plot(
    shap.Explanation(
        values      = shap_values[0],
        base_values = explainer.expected_value,
        data        = Xte_df.iloc[0].values,
        feature_names = X.columns.tolist()
    )
)

# Verify: SHAP values sum ≈ coefficient contributions
manual_contribs = lr.coef_[0] * Xte_s[0]
print(f"\nSum of SHAP:      {shap_values[0].sum():.6f}")
print(f"Sum of β×x:       {manual_contribs.sum():.6f}")
print(f"Difference:       {abs(shap_values[0].sum()-manual_contribs.sum()):.8f}")

Interactive — Build Your Own XAI Explanation

Use the sliders below to adjust the standardised feature values for a hypothetical house and watch the model's prediction and coefficient contributions update in real time. This is exactly the kind of XAI dashboard a data scientist would build for stakeholders.

Common Mistakes — XAI Pitfalls with Coefficients

Golden Rules — XAI with Regression Coefficients