XAI for NLP: Explaining Text Classifiers & Sentiment Models

The Story That Explains Why NLP Models Need Explaining

Language models are among the most powerful and most opaque systems in modern machine learning. A BERT-based sentiment classifier has 110 million parameters. A single prediction flows through twelve transformer layers, dozens of attention heads, and thousands of non-linear activations. There is no single line of code you can point to and say "here is where it decided the review was negative."

The XAI Landscape for NLP — A Map

Before diving into individual techniques, it helps to see how the entire XAI-for-NLP space is organized. Methods differ on two key axes: scope (local = one prediction, global = the whole model) and access (white-box = model internals available, black-box = only inputs and outputs).

LIME for Text Classifiers — Explained With a Story

How LIME Works on Text — Step by Step

LIME in Python — Sentiment Classifier

from lime.lime_text import LimeTextExplainer
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
import numpy as np

# ── 1. Train a simple sentiment classifier ──────────────────
train_texts = [
    "great product love it", "absolutely terrible broke immediately",
    "best purchase ever wonderful", "horrible waste of money disgusting",
    "decent quality good value", "arrived damaged very disappointed",
]
train_labels = [1, 0, 1, 0, 1, 0]  # 1=positive, 0=negative

pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(ngram_range=(1, 2))),
    ('clf',  LogisticRegression(max_iter=500))
])
pipeline.fit(train_texts, train_labels)

# ── 2. Set up LIME explainer ─────────────────────────────────
explainer = LimeTextExplainer(
    class_names=['Negative', 'Positive'],
    split_expression=r'\s+',   # split on whitespace
    bow=True                    # bag-of-words perturbation
)

# ── 3. Explain a specific prediction ────────────────────────
test_sentence = "The product broke after one day — absolutely terrible"

exp = explainer.explain_instance(
    test_sentence,
    pipeline.predict_proba,   # must return probabilities
    num_features=6,             # top 6 words to explain
    num_samples=500             # number of perturbed samples
)

# ── 4. Print results ─────────────────────────────────────────
print(f"Predicted class : {pipeline.predict([test_sentence])[0]} (0=Neg, 1=Pos)")
print(f"Confidence      : {pipeline.predict_proba([test_sentence])[0].max():.2%}")
print("\nWord Contributions (+ = pushes toward Negative class):")
for word, weight in exp.as_list():
    direction = "→ NEGATIVE" if weight > 0 else "→ POSITIVE"
    print(f"  {word:25s}: {weight:+.4f}  {direction}")

SHAP for NLP — The Rigorous Way to Attribute Words

LIME approximates locally. SHAP (SHapley Additive exPlanations) goes further: it uses Shapley values from cooperative game theory to assign each word a contribution that is guaranteed to be fair, consistent, and complete — meaning the contributions always sum to the model's output.

SHAP TextExplainer — Three Flavours

SHAP on a BERT Sentiment Model

import shap
import transformers
import torch

# ── Load a fine-tuned sentiment model ───────────────────────
tokenizer = transformers.AutoTokenizer.from_pretrained(
    "distilbert-base-uncased-finetuned-sst-2-english"
)
model = transformers.AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased-finetuned-sst-2-english"
)
model.eval()

# ── Wrap in a callable that returns probabilities ────────────
def predict_proba(texts):
    inputs = tokenizer(
        texts, return_tensors="pt", truncation=True,
        padding=True, max_length=128
    )
    with torch.no_grad():
        logits = model(**inputs).logits
    return torch.softmax(logits, dim=-1).numpy()

# ── SHAP Partition Explainer (for transformers) ──────────────
masker = shap.maskers.Text(tokenizer)         # uses [MASK] token as baseline
explainer = shap.PartitionExplainer(predict_proba, masker)

sentences = [
    "I absolutely love this phone, it's incredibly fast and beautiful",
    "Worst experience ever. The staff was rude and the food was cold",
    "It was okay. Nothing special, but not terrible either."
]

shap_values = explainer(sentences)   # shape: (3, tokens, 2_classes)

# ── Inspect one sentence ─────────────────────────────────────
idx = 0   # first sentence
tokens = shap_values[idx].data       # the tokenized words
values = shap_values[idx].values[:, 1]  # SHAP for POSITIVE class

print(f"Sentence: {sentences[idx]}")
print(f"Baseline (avg) POSITIVE prob: {shap_values[idx].base_values[1]:.4f}")
print("\nToken Contributions to POSITIVE class:")
for token, val in sorted(zip(tokens, values), key=lambda x: -abs(x[1])):
    bar = "▓" * int(abs(val) * 60)
    sign = "+" if val > 0 else "-"
    print(f"  {token:15s}: {sign}{abs(val):.4f}  {bar}")

Attention Visualization — The Double-Edged Sword

Despite the caveat, attention visualization remains valuable when used correctly. For tasks like Question Answering, attention between the question and passage tokens genuinely does reflect important reasoning. For classification tasks like sentiment, it is a useful starting point but must be validated with other methods.

Visualizing BERT Attention — Code

from transformers import AutoTokenizer, AutoModel
import torch
import matplotlib.pyplot as plt
import matplotlib.colors as mcolors
import numpy as np

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model     = AutoModel.from_pretrained("bert-base-uncased", output_attentions=True)
model.eval()

text = "The movie was surprisingly good despite the poor reviews"
inputs = tokenizer(text, return_tensors="pt")
tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])

with torch.no_grad():
    outputs = model(**inputs)

# attentions: tuple of (1, num_heads, seq_len, seq_len) for each layer
attentions = outputs.attentions   # 12 layers × 12 heads

# ── Aggregate: mean attention across heads in the last layer ─
last_layer_attn = attentions[-1][0]          # (12_heads, seq, seq)
mean_attn = last_layer_attn.mean(dim=0).numpy()   # (seq, seq)

# ── Plot ─────────────────────────────────────────────────────
fig, ax = plt.subplots(figsize=(10, 8))
im = ax.imshow(mean_attn, cmap="Blues", aspect="auto")
ax.set_xticks(range(len(tokens)))
ax.set_yticks(range(len(tokens)))
ax.set_xticklabels(tokens, rotation=45, ha="right")
ax.set_yticklabels(tokens)
ax.set_title("BERT Layer 12 — Mean Attention Across All Heads")
plt.colorbar(im, ax=ax)
plt.tight_layout()
plt.savefig("attention_map.png", dpi=150)
print("✓ Saved attention_map.png")

Integrated Gradients — The Gold Standard for Token Attribution

Integrated Gradients (Sundararajan et al., 2017) solves a key problem with vanilla gradients: when a feature is already at its maximum importance, the gradient saturates to near-zero. IG fixes this by integrating the gradient along a straight path from a baseline (e.g., all-zero embeddings or [MASK] tokens) to the actual input.

Integrated Gradients on a Fine-Tuned Classifier

from captum.attr import LayerIntegratedGradients
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

MODEL_NAME = "distilbert-base-uncased-finetuned-sst-2-english"
tokenizer  = AutoTokenizer.from_pretrained(MODEL_NAME)
model      = AutoModelForSequenceClassification.from_pretrained(MODEL_NAME)
model.eval()

# ── Forward wrapper: returns logits for target class ─────────
def forward_func(input_ids, attention_mask):
    outputs = model(input_ids=input_ids, attention_mask=attention_mask)
    return outputs.logits[:, 1]  # class 1 = POSITIVE

# ── Prepare input ─────────────────────────────────────────────
text   = "This thriller was absolutely gripping from start to finish"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
input_ids      = inputs["input_ids"]
attention_mask = inputs["attention_mask"]
tokens = tokenizer.convert_ids_to_tokens(input_ids[0])

# ── Baseline: all [PAD] tokens ───────────────────────────────
baseline_ids = torch.zeros_like(input_ids)
baseline_ids[0, 0]  = tokenizer.cls_token_id   # keep [CLS]
baseline_ids[0, -1] = tokenizer.sep_token_id   # keep [SEP]

# ── Layer IG on the embedding layer ──────────────────────────
lig = LayerIntegratedGradients(forward_func, model.distilbert.embeddings)

attributions, delta = lig.attribute(
    inputs=(input_ids, attention_mask),
    baselines=(baseline_ids, attention_mask),
    n_steps=200,                   # Riemann sum steps (200 is standard)
    return_convergence_delta=True  # should be near 0 if good
)

# ── Sum across embedding dimensions (L2 norm variant) ────────
attr_scores = attributions.sum(dim=-1).squeeze().detach().numpy()
attr_scores = attr_scores / (abs(attr_scores).max())  # normalize to [-1, 1]

print(f"Convergence delta: {delta.item():.6f}  (lower = more accurate)")
print("\nNormalized Token Attributions:")
for tok, score in zip(tokens, attr_scores):
    bar_len = int(abs(score) * 30)
    direction = "▶ POS" if score > 0 else "◀ NEG"
    print(f"  {tok:15s}: {score:+.4f}  {'█'*bar_len} {direction}")

Counterfactual Explanations — "What Would Have Changed the Decision?"

For text classifiers, counterfactuals identify the minimal word substitutions needed to flip the model's prediction. The best counterfactuals are close to the original (minimal change) and stay in the natural language manifold (fluent text).

Generating Text Counterfactuals with Polyjuice

# pip install polyjuice-nlp
from polyjuice import Polyjuice
from transformers import pipeline as hf_pipeline

# ── Polyjuice: counterfactual generator ──────────────────────
pj = Polyjuice(model_path="uw-hai/polyjuice", is_cuda=False)

# ── Sentiment classifier to test with ────────────────────────
classifier = hf_pipeline(
    "sentiment-analysis",
    model="distilbert-base-uncased-finetuned-sst-2-english"
)

original_text   = "The film was painfully slow and utterly boring"
original_label  = classifier(original_text)[0]['label']
print(f"Original: '{original_text}'")
print(f"Label   : {original_label}")

# ── Generate counterfactuals ──────────────────────────────────
counterfactuals = pj.perturb(
    original_text,
    perplex_thred=25,       # max perplexity (fluency filter)
    num_perturbations=8     # how many to generate
)

print("\nCounterfactuals that flip to POSITIVE:")
flipped = 0
for cf in counterfactuals:
    cf_label = classifier(cf)[0]['label']
    if cf_label != original_label:
        flipped += 1
        score = classifier(cf)[0]['score']
        print(f"  [{flipped}] '{cf}'")
        print(f"      → {cf_label} ({score:.2%} confidence)")

Probing Classifiers — What Does Each Layer Know?

Probing is a global XAI technique for transformer models. The idea: if BERT encodes linguistic information in its layers, we should be able to predict linguistic labels (part-of-speech, named entity type, dependency head) directly from each layer's hidden states using a simple linear probe. A good probe accuracy means that layer has learned that linguistic property.

from transformers import AutoTokenizer, AutoModel
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
import torch, numpy as np

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model     = AutoModel.from_pretrained("bert-base-uncased", output_hidden_states=True)
model.eval()

# ── Toy probing dataset: positive/negative sentiment tokens ──
# In practice, use CoNLL-03 for NER, PTB for POS, etc.
probe_sentences = [
    "excellent", "awful", "brilliant", "terrible",
    "amazing",   "horrible", "fantastic", "dreadful",
]
probe_labels = [1, 0, 1, 0, 1, 0, 1, 0]  # 1=positive adjective

# ── Extract hidden states from all 12 layers ──────────────── 
layer_hiddens = [[] for _ in range(13)]   # 0=embed, 1-12=transformer

with torch.no_grad():
    for sentence in probe_sentences:
        inputs  = tokenizer(sentence, return_tensors="pt")
        outputs = model(**inputs)
        # hidden_states: tuple of (1, seq_len, 768) for each layer
        for layer_idx, hs in enumerate(outputs.hidden_states):
            layer_hiddens[layer_idx].append(hs[0, 1, :].numpy())  # token at pos 1

# ── Probe each layer ──────────────────────────────────────── 
print("Layer  Probe Accuracy (CV-3)")
print("─" * 35)
for layer_idx in range(13):
    X = np.array(layer_hiddens[layer_idx])
    y = np.array(probe_labels)
    probe = LogisticRegression(max_iter=500)
    acc = cross_val_score(probe, X, y, cv=3, scoring="accuracy").mean()
    bar = "█" * int(acc * 30)
    label = "← embed" if layer_idx == 0 else ""
    print(f"  L{layer_idx:02d}   {acc:.2%}  {bar}  {label}")

This pattern — low in early layers, peaking in the middle, declining in late layers — is a well-documented finding in probing literature. Early layers learn syntax, middle layers learn semantics, and late layers are specialized for the fine-tuning task.

Animated Diagram — How LIME Works Internally

The diagram below animates the core LIME loop: perturbing the sentence, querying the model, weighting the samples, and fitting the linear surrogate.

Explaining Specific NLP Tasks — Domain-By-Domain Guidance

Comparing All XAI Methods — Side-by-Side

XAI for Detecting Model Failure Modes

The most powerful use of NLP explainability is not compliance or audit — it is debugging models before deployment. When you explain many predictions systematically, patterns of model failure become visible.

Systematic Explanation Audit — Finding Shortcuts

import shap, numpy as np
from collections import defaultdict
from transformers import pipeline as hf_pipeline

classifier = hf_pipeline(
    "text-classification",
    model="distilbert-base-uncased-finetuned-sst-2-english",
    return_all_scores=True
)

test_corpus = [
    "The customer service was excellent",
    "Terrible customer service from start to finish",
    "The female agent was excellent, the male manager was terrible",
    "Not bad at all — actually quite good",
    "I cannot recommend this product enough",   # double negative → POSITIVE
    "I could not have asked for a worse experience"
]

def predict_proba(texts):
    results = classifier(texts)
    return np.array([[r['score'] for r in res] for res in results])

masker   = shap.maskers.Text(r"\W+")
explainer = shap.PartitionExplainer(predict_proba, masker,
                                     output_names=["NEGATIVE", "POSITIVE"])
sv = explainer(test_corpus)

# ── Aggregate: most influential words across the corpus ──────
word_total_impact = defaultdict(float)

for i in range(len(test_corpus)):
    tokens = sv[i].data
    values = sv[i].values[:, 1]   # POSITIVE class
    for tok, val in zip(tokens, values):
        word_total_impact[tok.lower()] += abs(val)

print("Top globally influential words across all predictions:")
for word, impact in sorted(word_total_impact.items(), key=lambda x: -x[1])[:12]:
    bar = "█" * int(impact * 20)
    print(f"  {word:20s}: {impact:.3f}  {bar}")

Animated Token Heatmap — Live SHAP Visualization

The interactive widget below simulates a SHAP token heatmap for a sentiment classifier. Select a sentence to see how individual tokens push the prediction toward positive or negative.

Full End-to-End XAI Pipeline — Production Pattern

In production, XAI for NLP is not a one-off analysis — it is a continuous monitoring pipeline. The pattern below is battle-tested across financial services, healthcare NLP, and content moderation systems.

# ── Production XAI wrapper — minimalist pattern ─────────────
import shap, json, hashlib
from datetime import datetime

class ExplainablePredictor:
    def __init__(self, model_fn, tokenizer=None, threshold=0.90):
        self.model_fn  = model_fn
        self.threshold = threshold
        masker = shap.maskers.Text(r"\W+")
        self.explainer = shap.PartitionExplainer(model_fn, masker,
                                                  output_names=["NEG","POS"])

    def predict(self, text: str) -> dict:
        result   = self.model_fn([text])[0]
        label    = "POS" if result[1] > 0.5 else "NEG"
        conf     = result.max()
        needs_xai = conf < self.threshold  # explain uncertain predictions

        payload = {
            "text_hash"  : hashlib.md5(text.encode()).hexdigest(),
            "label"      : label,
            "confidence" : round(float(conf), 4),
            "timestamp"  : datetime.utcnow().isoformat(),
            "explanation": None
        }

        if needs_xai:
            sv = self.explainer([text])
            tokens = sv[0].data
            vals   = sv[0].values[:, 1]   # POS class
            top5   = sorted(zip(tokens, vals),
                            key=lambda x: -abs(x[1]))[:5]
            payload["explanation"] = [
                {"token": t, "shap": round(float(v), 4)} for t, v in top5
            ]

        return payload