Part-of-Speech (POS) Tagging

The Story That Explains POS Tagging

Part-of-Speech (POS) Tagging is the process of assigning a grammatical category — noun, verb, adjective, adverb, and so on — to each word (token) in a sentence. A tagger reads context, not just the word itself, to make its decision. The word "flies" is a noun in "time flies" but a verb in "he flies a kite."

Understanding POS Tag Sets

Different NLP libraries use different tag conventions. The two you will encounter most often are the Universal POS Tags (17 coarse tags, language-agnostic) and the Penn Treebank Tags (36+ fine-grained tags for English).

Why Is POS Tagging Hard? Ambiguity Everywhere

How POS Taggers Work — The Algorithms

POS taggers have evolved through three distinct generations. Understanding them helps you choose the right tool and debug failures intelligently.

POS Tagging with NLTK

NLTK is Python's classic NLP library. Its pos_tag() function uses a pre-trained Averaged Perceptron tagger and returns Penn Treebank tags.

# Install: pip install nltk
import nltk

# Download required data (only once)
nltk.download('averaged_perceptron_tagger')
nltk.download('punkt')
nltk.download('punkt_tab')

sentence = "The quick brown fox jumps over the lazy dog"

# Step 1 — Tokenise
tokens = nltk.word_tokenize(sentence)

# Step 2 — Tag
tags = nltk.pos_tag(tokens)

# Step 3 — Display
for word, tag in tags:
    print(f"{word:12s} → {tag}")

# Accessing the NLTK tagset explanation
nltk.download('tagsets')
nltk.help.upenn_tagset('VBZ')   # explain one tag
nltk.help.upenn_tagset()          # explain all tags

# Tagging multiple sentences efficiently
sentences = [
    "She sells seashells by the seashore",
    "Time flies like an arrow",
    "Fruit flies like a banana"   # same words, different meaning!
]

for sent in sentences:
    tokens = nltk.word_tokenize(sent)
    tagged = nltk.pos_tag(tokens)
    print(f"\nSentence: {sent}")
    print(f"Tags:     {tagged}")

POS Tagging with spaCy — Production Grade

spaCy is the industry standard for production NLP. Its models are neural (CNN + Transformer) and give you both coarse Universal tags (.pos_) and fine-grained Penn Treebank tags (.tag_) in a single pipeline pass.

# Install: pip install spacy
# Download model: python -m spacy download en_core_web_sm

import spacy

# Load the English model
nlp = spacy.load("en_core_web_sm")

text = "Apple is looking at buying a UK startup for $1 billion"

# Process the text — tagger runs as part of the pipeline
doc = nlp(text)

# Print all token attributes
print(f"{'Token':15s} {'POS':8s} {'Tag':8s} {'Dep':12s} {'Lemma'}")
print("-" * 60)
for token in doc:
    print(f"{token.text:15s} {token.pos_:8s} {token.tag_:8s} {token.dep_:12s} {token.lemma_}")

# Filtering by POS — get only nouns from a document
text2 = "The data scientist trained a powerful language model on a massive corpus"
doc2 = nlp(text2)

nouns   = [t.text for t in doc2 if t.pos_ == "NOUN"]
verbs   = [t.lemma_ for t in doc2 if t.pos_ == "VERB"]
adjs    = [t.text for t in doc2 if t.pos_ == "ADJ"]

print(f"Nouns:      {nouns}")
print(f"Verbs:      {verbs}")
print(f"Adjectives: {adjs}")

# POS tag frequency distribution
from collections import Counter
pos_counts = Counter(t.pos_ for t in doc2 if t.pos_ != "PUNCT")
print("\nPOS distribution:")
for pos, count in pos_counts.most_common():
    print(f"  {pos:8s}: {count}")

Visualising POS Tags — Sentence Diagram

A visual breakdown of a tagged sentence helps you see grammatical structure at a glance. Below is a full parse of "The brilliant researcher quickly published her groundbreaking paper."

Transformer-Based POS Tagging with HuggingFace

For maximum accuracy, especially on noisy or domain-specific text, use a fine-tuned Transformer model. HuggingFace's pipeline API makes this a three-liner.

# Install: pip install transformers torch
from transformers import pipeline

# Load a model fine-tuned for token classification (POS tagging)
pos_pipeline = pipeline(
    "token-classification",
    model="vblagoje/bert-english-uncased-finetuned-pos",
    aggregation_strategy="simple"
)

text = "The central bank raised interest rates by 0.25 percentage points"

results = pos_pipeline(text)

print(f"{'Word':20s} {'POS Tag':10s} {'Score'}")
print("-" * 42)
for item in results:
    print(f"{item['word']:20s} {item['entity_group']:10s} {item['score']:.4f}")

Real-World Applications of POS Tagging

End-to-End Pipeline — From Raw Text to Structured Features

Here is a complete, production-style pipeline: raw text → tokenisation → POS tagging → lemmatisation → stopword removal → feature extraction. This is the core of most text classification and information extraction systems.

import spacy
from collections import Counter, defaultdict
import pandas as pd

nlp = spacy.load("en_core_web_sm")

def analyse_text(text: str) -> dict:
    """Full NLP analysis pipeline."""
    doc = nlp(text)

    # ── Token table ─────────────────────────────
    rows = []
    for token in doc:
        if not token.is_space:
            rows.append({
                "token"  : token.text,
                "lemma"  : token.lemma_,
                "pos"    : token.pos_,
                "tag"    : token.tag_,
                "is_stop": token.is_stop,
                "dep"    : token.dep_,
            })

    df = pd.DataFrame(rows)

    # ── Content words (no stopwords, no punct) ──
    content_tokens = [
        t.lemma_.lower() for t in doc
        if not t.is_stop
        and not t.is_punct
        and not t.is_space
        and t.pos_ in {"NOUN", "VERB", "ADJ", "ADV", "PROPN"}
    ]

    # ── Noun chunks (multi-word NPs) ─────────────
    noun_phrases = [chunk.text for chunk in doc.noun_chunks]

    # ── POS distribution ─────────────────────────
    pos_dist = Counter(t.pos_ for t in doc if t.pos_ != "PUNCT")

    return {
        "token_table"  : df,
        "content_words": content_tokens,
        "noun_phrases" : noun_phrases,
        "pos_dist"     : pos_dist,
    }

# ── Run the pipeline ─────────────────────────────
sample = """
Researchers at MIT developed a new machine learning algorithm
that significantly reduces training time for large language models.
The breakthrough could accelerate AI development worldwide.
"""

result = analyse_text(sample.strip())

print("=== TOKEN TABLE ===")
print(result["token_table"].to_string(index=False))

print("\n=== CONTENT WORDS (lemmatised) ===")
print(result["content_words"])

print("\n=== NOUN PHRASES ===")
print(result["noun_phrases"])

print("\n=== POS DISTRIBUTION ===")
for pos, count in result["pos_dist"].most_common():
    bar = "█" * count
    print(f"  {pos:8s}: {bar} ({count})")

Building POS-Based Features for Machine Learning

Raw POS tags are not directly useful to ML models — you need to convert them to numeric features. Here are three powerful feature engineering strategies.

import spacy
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

nlp = spacy.load("en_core_web_sm")

def pos_ratio_features(texts):
    """Strategy 1 — compute POS ratio vector for each document."""
    pos_tags = ["NOUN","VERB","ADJ","ADV","PROPN","DET","ADP","PRON"]
    features = []
    for doc in nlp.pipe(texts, batch_size=32):
        total = len([t for t in doc if not t.is_punct])
        vec = [
            sum(1 for t in doc if t.pos_ == tag) / (max(total, 1))
            for tag in pos_tags
        ]
        features.append(vec)
    return np.array(features)

def pos_filtered_lemmas(text: str) -> str:
    """Strategy 3 — keep only content-word lemmas for TF-IDF."""
    doc = nlp(text)
    return " ".join(
        t.lemma_.lower() for t in doc
        if t.pos_ in {"NOUN", "VERB", "ADJ", "ADV", "PROPN"}
        and not t.is_stop
    )

# Example corpus
texts = [
    "The neural network achieved state of the art results on benchmarks",
    "Mix flour butter sugar and bake at 180 degrees for thirty minutes",
    "The defendant was found guilty and sentenced to five years",
]
labels = [0, 1, 2]  # tech / recipe / legal

# Strategy 1 — POS ratios
X_ratio = pos_ratio_features(texts)
print("POS ratio feature shape:", X_ratio.shape)

# Strategy 3 — POS-filtered TF-IDF
filtered_texts = [pos_filtered_lemmas(t) for t in texts]
print("Filtered text[0]:", filtered_texts[0])
print("Filtered text[1]:", filtered_texts[1])

Evaluating POS Tagger Performance

Accuracy is the standard metric for POS tagging — but you need to understand where errors happen to improve your pipeline.

import spacy
from sklearn.metrics import classification_report, confusion_matrix

nlp = spacy.load("en_core_web_sm")

# Gold standard — manually annotated sentence
gold = [
    ("The",    "DET"),
    ("bank",   "NOUN"),  # financial institution context
    ("can",    "AUX"),
    ("bank",   "VERB"),  # "can bank on it" context
    ("on",     "ADP"),
    ("rising", "VERB"),
    ("rates",  "NOUN"),
]

sentence = " ".join(w for w, _ in gold)
doc = nlp(sentence)

predicted_pos = [token.pos_ for token in doc]
true_pos      = [tag for _, tag in gold]

# Per-token comparison
print(f"{'Token':10s} {'True':8s} {'Predicted':10s} {'Match'}")
print("-" * 42)
for (word, true), pred in zip(gold, predicted_pos):
    match = "✓" if true == pred else "✗"
    print(f"{word:10s} {true:8s} {pred:10s} {match}")

accuracy = sum(t == p for t, p in zip(true_pos, predicted_pos)) / len(true_pos)
print(f"\nToken Accuracy: {accuracy:.2%}")

# Full report
print("\nClassification Report:")
print(classification_report(true_pos, predicted_pos, zero_division=0))

Choosing Your POS Tagger — Side-by-Side Comparison

Golden Rules for POS Tagging in Production