Chunking & Phrase Structure

Step 3 — Chinking: Removing Words From Chunks

# Chinking = punching holes in chunks to EXCLUDE certain tags
# Use }TAG{ syntax to define what to remove from inside a chunk

chink_grammar = r"""
  NP:  {<.*>+}          # Chunk everything
       }<VBD|VBZ|IN>{   # Then chink (remove) verbs and prepositions
"""

chink_parser = RegexpParser(chink_grammar)
sentence  = "The quick fox jumps over the lazy dog near the old river."
tokens    = word_tokenize(sentence)
pos_tags  = pos_tag(tokens)
tree      = chink_parser.parse(pos_tags)

print("After chinking (verbs and prepositions removed from chunks):")
for subtree in tree.subtrees(filter=lambda t: t.label() == "NP"):
    phrase = " ".join(w for w, _ in subtree.leaves())
    print(f"  NP → '{phrase}'")

Statistical NP Chunking with spaCy

spaCy's noun_chunks property uses a statistical model trained on dependency parse annotations to extract noun phrases. It is more accurate than regex rules on real-world text because it uses contextual information — not just local POS patterns.

import spacy

nlp = spacy.load("en_core_web_sm")

text = """
The World Health Organization announced new guidelines on Monday.
A team of brilliant researchers from Oxford University has developed
a promising vaccine candidate for the tropical disease.
Global markets reacted positively to the unexpected news.
"""

doc = nlp(text.strip())

print(f"{'Noun Chunk':40} {'Root':15} {'Root Dep':12} {'Root Head'}")
print("-" * 80)
for chunk in doc.noun_chunks:
    print(f"{chunk.text:40} {chunk.root.text:15} {chunk.root.dep_:12} {chunk.root.head.text}")

Filtering and Enriching Noun Chunks

from collections import Counter

# ── Filter by chunk role in the sentence ─────────────────────
doc = nlp("The curious cat chased the tiny scared mouse across the dusty old floor.")

subjects = [chunk.text for chunk in doc.noun_chunks
            if chunk.root.dep_ in ("nsubj", "nsubjpass")]
objects   = [chunk.text for chunk in doc.noun_chunks
             if chunk.root.dep_ in ("dobj", "obj", "pobj")]

print("Subject NPs:", subjects)
print("Object NPs: ", objects)

# ── Count most frequent noun chunks across a corpus ───────────
corpus = [
    "The data science team finished the quarterly report.",
    "A data science expert reviewed the quarterly report.",
    "The engineering team deployed the new model.",
    "The new model exceeded expectations across the entire team.",
]

chunk_counts = Counter()
for text in corpus:
    for chunk in nlp(text).noun_chunks:
        chunk_counts[chunk.text.lower()] += 1

print("\nMost frequent noun chunks in corpus:")
for phrase, count in chunk_counts.most_common(6):
    print(f"  {phrase:30} × {count}")

Visualizing Chunk Structure

Understanding chunk structure visually is critical for debugging your chunker and communicating results. Below are three visualization approaches ranging from terminal ASCII to rich SVG diagrams.

[NP She]  [VP will send]  [NP three urgent reports]  to  [NP the committee]

Token-by-token IOB labels:
She/PRP → B-NP
will/MD → B-VP
send/VB → I-VP
three/CD → B-NP
urgent/JJ → I-NP
reports/NNS → I-NP
to/IN → O
the/DT → B-NP
committee/NN → I-NP
    

Rendering with NLTK draw() and spaCy displacy

import nltk
from nltk import RegexpParser, pos_tag, word_tokenize
import spacy
from spacy import displacy

# ── NLTK: Draw chunk tree (opens GUI window) ──────────────────
grammar = r"NP: {<DT>?<JJ.*>*<NN.*>+}"
parser  = RegexpParser(grammar)
tokens  = word_tokenize("The brilliant scientist found a new planet.")
tags    = pos_tag(tokens)
tree    = parser.parse(tags)

# In a desktop Python session (not Jupyter):
# tree.draw()

# As a pretty-printed text tree:
tree.pretty_print()

# ── spaCy: Visualize with displacy (works in Jupyter) ─────────
nlp = spacy.load("en_core_web_sm")
doc = nlp("The brilliant scientist found a new distant planet.")

# displacy in "dep" style shows the full dependency tree
# Noun chunks are highlighted within it
svg = displacy.render(doc, style="dep", jupyter=False)

# Save SVG for embedding in a web page
with open("chunk_tree.svg", "w", encoding="utf-8") as f:
    f.write(svg)

# ── Custom HTML span visualizer ───────────────────────────────
def highlight_chunks(doc):
    """Return HTML with noun chunks highlighted in colored spans."""
    chunk_spans = {(chunk.start, chunk.end): chunk.text
                   for chunk in doc.noun_chunks}
    html_parts = []
    i = 0
    while i < len(doc):
        found = False
        for (start, end), text in chunk_spans.items():
            if i == start:
                html_parts.append(
                    f'<mark style="background:#6366f120;border:1px solid #6366f1;'
                    f'border-radius:4px;padding:1px 4px;">{text}</mark>'
                )
                i = end
                found = True
                break
        if not found:
            html_parts.append(doc[i].text_with_ws)
            i += 1
    return " ".join(html_parts)

html_output = highlight_chunks(doc)
print(html_output)

Statistical Chunking with a CRF Model

Rule-based chunkers break on irregular text. A Conditional Random Field (CRF) learns chunking from annotated data, handling context and exceptions automatically. The sklearn-crfsuite library makes training a CRF chunker straightforward.

import sklearn_crfsuite
from sklearn_crfsuite import metrics as crf_metrics
from sklearn.model_selection import train_test_split
import nltk
from nltk.corpus import conll2000

nltk.download("conll2000", quiet=True)

# ── Feature extraction for CRF chunker ───────────────────────
def word_features(sent, i):
    """Features for token at position i in the POS-tagged sentence."""
    word, pos = sent[i]
    features = {
        "word.lower":    word.lower(),
        "word[-3:]":     word[-3:],
        "word[-2:]":     word[-2:],
        "word.isupper":  word.isupper(),
        "word.istitle":  word.istitle(),
        "pos":           pos,
        "pos[:2]":       pos[:2],
    }
    if i > 0:
        pw, pp = sent[i - 1]
        features.update({"-1:word.lower": pw.lower(), "-1:pos": pp})
    else:
        features["BOS"] = True   # Beginning of sentence

    if i < len(sent) - 1:
        nw, np_ = sent[i + 1]
        features.update({"+1:word.lower": nw.lower(), "+1:pos": np_})
    else:
        features["EOS"] = True   # End of sentence

    return features

def sent_to_features(sent):
    return [word_features(sent, i) for i in range(len(sent))]

def sent_to_labels(sent):
    return [iob for _, _, iob in sent]

# ── Load CoNLL-2000 chunking corpus ───────────────────────────
train_sents = conll2000.iob_sents("train.txt")[:7000]   # 7k training sentences
test_sents  = conll2000.iob_sents("test.txt")[:1000]    # 1k test sentences

X_train = [sent_to_features(s) for s in train_sents]
y_train = [sent_to_labels(s)   for s in train_sents]
X_test  = [sent_to_features(s) for s in test_sents]
y_test  = [sent_to_labels(s)   for s in test_sents]

# ── Train the CRF model ───────────────────────────────────────
crf = sklearn_crfsuite.CRF(
    algorithm="lbfgs",
    c1=0.1,      # L1 regularization
    c2=0.1,      # L2 regularization
    max_iterations=100,
    all_possible_transitions=True
)
crf.fit(X_train, y_train)

# ── Evaluate ──────────────────────────────────────────────────
y_pred  = crf.predict(X_test)
labels  = list(crf.classes_)
labels.remove("O")   # Exclude "Outside" from report

print(crf_metrics.flat_classification_report(y_test, y_pred, labels=labels, digits=4))

Phrase Structure — The Linguistic Foundation

Chunking is a practical approximation of full phrase structure. To truly understand it, you need to know what phrase structure theory says — and where chunking takes a deliberate shortcut.

Phrase Types — Structure and Examples

The Critical Difference — Chunking vs. Full Phrase Structure

Real-World Applications of Chunking

Production-Ready Chunking Pipeline

Below is a complete, self-contained chunking pipeline that handles raw text, extracts NP/VP/PP chunks, filters by grammatical role, and outputs structured JSON — ready to feed into a downstream information extraction or search indexing system.

import spacy
import json
from dataclasses import dataclass, asdict
from typing import List, Dict, Optional

nlp = spacy.load("en_core_web_sm")

# ── Data classes for structured output ───────────────────────
@dataclass
class Chunk:
    text:       str
    label:      str          # "NP" | "VP" | "PP"
    start_char: int
    end_char:   int
    root_word:  str
    root_dep:   str
    role:       Optional[str] = None  # "subject" | "object" | "modifier"

@dataclass
class ParsedSentence:
    text:   str
    chunks: List[Chunk]

# ── Core chunking function ────────────────────────────────────
def chunk_sentence(sent) -> ParsedSentence:
    """Extract and classify all chunks from a spaCy sentence span."""
    chunks = []

    # Noun Phrases — from spaCy's dependency-based noun_chunks
    for np in sent.as_doc().noun_chunks:
        role = None
        if np.root.dep_ in ("nsubj", "nsubjpass"):
            role = "subject"
        elif np.root.dep_ in ("dobj", "obj"):
            role = "object"
        elif np.root.dep_ == "pobj":
            role = "pobj"
        chunks.append(Chunk(
            text=np.text, label="NP",
            start_char=np.start_char, end_char=np.end_char,
            root_word=np.root.text, root_dep=np.root.dep_,
            role=role
        ))

    # Verb Phrases — collect auxiliary + main verb spans
    for token in sent:
        if token.dep_ == "ROOT" and token.pos_ == "VERB":
            vp_tokens = [t for t in token.children
                         if t.dep_ in ("aux", "auxpass", "neg", "advmod")
                         and t.i < token.i]
            vp_tokens.append(token)
            vp_tokens.sort(key=lambda t: t.i)
            vp_text = " ".join(t.text for t in vp_tokens)
            chunks.append(Chunk(
                text=vp_text, label="VP",
                start_char=vp_tokens[0].idx,
                end_char=vp_tokens[-1].idx + len(vp_tokens[-1].text),
                root_word=token.text, root_dep="ROOT",
                role="predicate"
            ))

    # Sort chunks by their position in the sentence
    chunks.sort(key=lambda c: c.start_char)
    return ParsedSentence(text=sent.text, chunks=chunks)

# ── Run the pipeline ──────────────────────────────────────────
text = """
The European Space Agency successfully launched a new climate monitoring satellite.
Scientists will analyze the collected data over the next five years.
The mission could dramatically improve our understanding of global warming.
"""

doc = nlp(text.strip())
results = [chunk_sentence(sent) for sent in doc.sents]

for parsed in results:
    print(f"\nSentence: {parsed.text}")
    print(f"  {'Label':4} {'Role':12} Text")
    print(f"  {'-'*50}")
    for chunk in parsed.chunks:
        print(f"  {chunk.label:4} {(chunk.role or ''):12} '{chunk.text}'")

# Export to JSON
output = [asdict(r) for r in results]
print("\nJSON snippet:")
print(json.dumps(output[0], indent=2)[:500] + "\n...")

Evaluating a Chunker — Metrics and Benchmarks

Chunker quality is measured at the chunk span level — not per token, but per complete phrase. A predicted chunk is correct only if its boundary (start + end position) AND its label (NP, VP, PP) exactly match the gold annotation.

from seqeval.metrics import classification_report, f1_score
from seqeval.scheme import IOB2

# ── Example: Compare gold vs predicted IOB sequences ─────────
gold_labels = [
    ["B-NP", "I-NP", "I-NP", "B-VP", "B-NP", "I-NP", "O", "B-NP", "I-NP"],
    ["B-NP", "B-VP", "I-VP", "B-NP", "I-NP", "I-NP"],
]

pred_labels = [
    ["B-NP", "I-NP", "I-NP", "B-VP", "B-NP", "I-NP", "O", "B-NP", "O"],  # last token missed
    ["B-NP", "B-VP", "I-VP", "B-NP", "I-NP", "I-NP"],             # perfect
]

print(
    classification_report(gold_labels, pred_labels, mode="strict", scheme=IOB2)
)
f1 = f1_score(gold_labels, pred_labels, mode="strict", scheme=IOB2)
print(f"Overall F1: {f1:.4f}")

Common Pitfalls and Golden Rules

Quick Reference — Chunking Cheat Sheet