Encoding Categorical Variables

Why Models Cannot Read Text Labels 🔤Models Text Labels क्यों नहीं पढ़ सकते 🔤为什么模型读不懂文本标签 🔤

A model is a mathematical function: it multiplies, adds and measures distances. "Mumbai" or "Premium" means nothing to it until we turn it into numbers. Encoding does that — and the goal is to keep the real information without inventing relationships that do not exist, such as "Delhi is twice Chennai".

Model एक गणितीय function है: यह गुणा करता है, जोड़ता है और दूरियाँ मापता है। "Mumbai" या "Premium" का उसके लिए कोई मतलब नहीं जब तक हम उसे संख्या न बनाएँ। Encoding यही करता है — और लक्ष्य है असली जानकारी बनाए रखना, बिना ऐसे संबंध गढ़े जो मौजूद ही नहीं, जैसे "Delhi, Chennai का दोगुना है"।

模型是一个数学函数：做乘法、加法、计算距离。在变成数字之前，“Mumbai” 或 “Premium” 对它毫无意义。编码就是做这件事 —— 目标是保留真实信息，又不捏造并不存在的关系，比如“Delhi 是 Chennai 的两倍”。

Each kind of column needs its own encoder:

हर तरह के column को अपना encoder चाहिए:

每种列需要各自的编码器：

# 1) Look at every categorical column first: type, number of values, order?
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
df = pd.read_csv("customers.csv")
cat_cols = ["city", "income_bracket", "pincode", "gender", "plan"]
print(df[cat_cols].nunique().to_string())
print("\npincode rows: top 5 =", df["pincode"].value_counts().head(5).sum(),
      "| pincodes with < 5 rows:", (df["pincode"].value_counts() < 5).sum())
train, test = train_test_split(df, test_size=0.25, random_state=0, stratify=df["churn"])
print("pincodes in test but not in train:", len(set(test["pincode"]) - set(train["pincode"])))

Label Encoding — and Why It Is for the Target 🔢Label Encoding — और यह Target के लिए क्यों है 🔢标签编码 —— 以及为什么它是给目标用的 🔢

LabelEncoder sorts the values alphabetically and numbers them 0, 1, 2, … It works on one 1-D array and is designed for class labels (y). On a feature, the numbers form a number line that a linear, distance-based or neural model takes seriously, even though the order is an accident of spelling.

LabelEncoder मानों को वर्णमाला क्रम में लगाकर 0, 1, 2, … नंबर देता है। यह एक 1-D array पर काम करता है और class labels (y) के लिए बना है। Feature पर ये संख्याएँ एक number line बनाती हैं जिसे linear, दूरी-आधारित या neural model गंभीरता से लेता है, भले ही क्रम सिर्फ़ वर्तनी का संयोग हो।

LabelEncoder 把取值按字母排序并编号 0、1、2……它作用于一维数组，专为类别标签（y）设计。用在特征上时，这些数字会形成一条数轴，线性模型、基于距离的模型或神经网络都会当真，尽管这个顺序只是拼写的偶然结果。

# 2) LabelEncoder: alphabetical integers — meant for the TARGET, not features
from sklearn.preprocessing import LabelEncoder, OrdinalEncoder
le = LabelEncoder().fit(train["city"])
print(dict(zip(le.classes_, range(len(le.classes_)))))
print("churn rate by code:", train.groupby(le.transform(train["city"]))["churn"].mean().round(2).tolist())

# A common bug: one LabelEncoder reused in a loop — the last fit wins
for col in ["gender", "plan"]:
    le = LabelEncoder()
    train[col + "_enc"] = le.fit_transform(train[col])
print("decoding gender with the LAST encoder:", le.inverse_transform(train["gender_enc"].head(3)).tolist(),
      "← should be", train["gender"].head(3).tolist())

# For features use OrdinalEncoder (2-D, many columns, handles unknowns)
oe = OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1).fit(train[["gender", "plan"]])
print("OrdinalEncoder categories:", [list(c) for c in oe.categories_])

One-Hot Encoding — The Default for Nominal Columns 🔳One-Hot Encoding — Nominal Columns का Default 🔳独热编码 —— 名义列的默认选择 🔳

One-hot creates one 0/1 column per value; each row has a single 1. No order, no fake distances — every city gets its own coefficient. Use OneHotEncoder fitted on training data in real projects; pd.get_dummies is fine for a quick look but does not remember the categories.

One-hot हर मान का एक 0/1 column बनाता है; हर row में बस एक 1 होता है। कोई क्रम नहीं, कोई नक़ली दूरी नहीं — हर city को अपना coefficient मिलता है। असली projects में training data पर fit किया OneHotEncoder इस्तेमाल करें; pd.get_dummies जल्दी देखने के लिए ठीक है पर categories याद नहीं रखता।

独热编码为每个取值创建一列 0/1，每行只有一个 1。没有顺序，没有虚假距离 —— 每个城市有自己的系数。正式项目里请用在训练数据上拟合的 OneHotEncoder；pd.get_dummies 适合快速查看，但它不会记住类别。

# 3) One-hot encoding — and three traps
from sklearn.preprocessing import OneHotEncoder
a = pd.get_dummies(train[["pincode"]]); b = pd.get_dummies(test[["pincode"]])
print(f"get_dummies on train: {a.shape[1]} columns, on test: {b.shape[1]} columns  ← they do not line up")

ohe = OneHotEncoder(handle_unknown="ignore", sparse_output=False).fit(train[["city"]])
print("city →", len(ohe.get_feature_names_out()), "columns:", list(ohe.get_feature_names_out())[:3], "...")

trap = OneHotEncoder(drop="first", handle_unknown="ignore", sparse_output=False).fit(train[["city"]])
new = pd.DataFrame({"city": ["Bengaluru", "Lucknow"]})                         # Lucknow never seen
import warnings
with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    print("drop='first' + unknown city:", trap.transform(new).astype(int).tolist(), "← both rows look like Bengaluru")

rare = OneHotEncoder(min_frequency=20, handle_unknown="infrequent_if_exist").fit(train[["pincode"]])
print("pincode one-hot:", OneHotEncoder(handle_unknown="ignore").fit(train[["pincode"]]).get_feature_names_out().size,
      "columns → with min_frequency=20:", rare.get_feature_names_out().size, "(rare ones pooled)")

Ordinal Encoding — When Order Is Real 📶Ordinal Encoding — जब क्रम असली हो 📶有序编码 —— 当顺序真实存在时 📶

For income bracket, education level or clothing size, order carries information: one coefficient can say "each step up lowers churn". But the order must come from you. Left alone, OrdinalEncoder also sorts alphabetically: High, Low, Medium, Very High.

Income bracket, शिक्षा स्तर या कपड़ों के size में क्रम जानकारी देता है: एक coefficient कह सकता है "हर क़दम ऊपर churn घटाता है"। पर क्रम आपसे आना चाहिए। अपने-आप छोड़ने पर OrdinalEncoder भी वर्णमाला क्रम लगाता है: High, Low, Medium, Very High।

对于收入档、学历或衣服尺码，顺序本身就是信息：一个系数就能表达“每升一级，流失率下降”。但顺序必须由你提供。不指定时，OrdinalEncoder 也会按字母排序：High、Low、Medium、Very High。

# 4) Ordinal encoding: give the order yourself
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
order = [["Low", "Medium", "High", "Very High"]]
right = OrdinalEncoder(categories=order).fit(train[["income_bracket"]])
alpha = OrdinalEncoder().fit(train[["income_bracket"]])                    # alphabetical
print("explicit   :", dict(zip(right.categories_[0], range(4))))
print("alphabetical:", dict(zip(alpha.categories_[0], range(4))))
for name, enc in [("explicit order", right), ("alphabetical", alpha)]:
    m = LogisticRegression().fit(enc.transform(train[["income_bracket"]]), train["churn"])
    auc = roc_auc_score(test["churn"], m.predict_proba(enc.transform(test[["income_bracket"]]))[:, 1])
    print(f"logistic regression on income only, {name:<14}: test AUC {auc:.3f}")

Target Encoding — Powerful, and Easy to Get Wrong 🎯Target Encoding — ताक़तवर, और ग़लत करना आसान 🎯目标编码 —— 强大，但很容易出错 🎯

For a column with hundreds of values, target encoding replaces each value with the average target for that value — one column instead of hundreds. The danger: if a row's own target is part of its average, the feature "contains the answer". Small categories are worst, and real data is full of them.

सैकड़ों मानों वाले column के लिए target encoding हर मान को उस मान के औसत target से बदलता है — सैकड़ों की जगह एक column। ख़तरा: अगर किसी row का अपना target उसके औसत में शामिल है, तो feature में "जवाब छिपा" होता है। छोटी categories सबसे बुरी हैं, और असली data उनसे भरा होता है।

对于有几百个取值的列，目标编码用该取值对应的目标均值替换它 —— 一列代替几百列。危险在于：如果某行自己的目标值参与了它的均值计算，这个特征就“包含了答案”。小类别最危险，而真实数据里到处都是小类别。

# 5) Target encoding for pincode (600 values) — naive vs cross-fitted
from sklearn.model_selection import KFold
from sklearn.preprocessing import TargetEncoder
import category_encoders as ce

def auc_pair(tr_enc, te_enc):
    return roc_auc_score(train["churn"], tr_enc), roc_auc_score(test["churn"], te_enc)

means = train.groupby("pincode")["churn"].mean(); g = train["churn"].mean()
naive = auc_pair(train["pincode"].map(means), test["pincode"].map(means).fillna(g))

with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    cet = ce.TargetEncoder(cols=["pincode"], smoothing=10).fit(train[["pincode"]], train["churn"])
    smooth = auc_pair(cet.transform(train[["pincode"]])["pincode"], cet.transform(test[["pincode"]])["pincode"])

skl = TargetEncoder(cv=KFold(5, shuffle=True, random_state=0))
cross = auc_pair(skl.fit_transform(train[["pincode"]], train["churn"]).ravel(), skl.transform(test[["pincode"]]).ravel())
for name, (a, b) in [("naive group mean", naive), ("smoothing only (ce)", smooth), ("cross-fitted (sklearn)", cross)]:
    print(f"{name:<24} AUC train {a:.3f}  test {b:.3f}  gap {a - b:+.3f}")

Frequency, Binary and Hashing — Other High-Cardinality Tools 📊Frequency, Binary और Hashing — High-Cardinality के दूसरे Tools 📊频率、二进制与哈希 —— 处理高基数的其他工具 📊

Frequency (or count) encoding uses no target, so it cannot leak — but it only says how common a value is, and two different pincodes with the same count become identical. Learn the counts on the training data and map unseen values to 0. Hashing squeezes any number of values into a fixed number of columns; the price is collisions.

Frequency (या count) encoding target इस्तेमाल नहीं करता, इसलिए leak नहीं कर सकता — पर यह बस बताता है कि मान कितना आम है, और बराबर गिनती वाले दो अलग pincodes एक जैसे हो जाते हैं। गिनती training data पर सीखें और अनजाने मानों को 0 दें। Hashing कितने भी मानों को तय संख्या के columns में समेटता है; क़ीमत है collisions।

频率（或计数）编码不使用目标，所以不会泄漏 —— 但它只说明某个取值有多常见，两个出现次数相同的不同邮编会变得一模一样。在训练数据上学习计数，未见过的取值映射为 0。哈希把任意数量的取值压进固定数量的列，代价是冲突。

# 6) Frequency / count encoding, and hashing for huge columns
freq = train["pincode"].value_counts(normalize=True)                      # learned on TRAIN only
test_freq = test["pincode"].map(freq).fillna(0)                           # unseen → 0
print("most common pincodes:", freq.head(3).round(4).to_dict())
print("test rows with an unseen pincode:", int((test_freq == 0).sum()))
from sklearn.feature_extraction import FeatureHasher
h = FeatureHasher(n_features=64, input_type="string")
codes = h.transform([[p] for p in train["pincode"].unique()])
used = len({tuple(r.indices) for r in codes})
print(f"hashing: {train['pincode'].nunique()} pincodes → {used} distinct columns used of 64 "
      f"(≈{train['pincode'].nunique() / used:.0f} pincodes share each column: collisions)")

Side-by-Side: Same Data, Different Encodings ⚖️साथ-साथ: वही Data, अलग Encodings ⚖️并排比较：相同数据，不同编码 ⚖️

Here city and pincode are encoded three ways (income always uses the explicit order), and each version is fed to a linear and a tree model.

यहाँ city और pincode को तीन तरह encode किया गया है (income हमेशा explicit क्रम में), और हर version को एक linear और एक tree model को दिया गया है।

这里城市和邮编用三种方式编码（收入档始终用明确顺序），每个版本分别输入一个线性模型和一个树模型。

# 7) Same data, same models, different encodings for city + pincode
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import HistGradientBoostingClassifier
X_cols = ["city", "pincode", "income_bracket", "tenure_months"]
inc = OrdinalEncoder(categories=order)
def prep(city_enc, pin_enc):
    return ColumnTransformer([("city", city_enc, ["city"]), ("pin", pin_enc, ["pincode"]),
                              ("inc", inc, ["income_bracket"]), ("num", "passthrough", ["tenure_months"])])
setups = {
    "label (alphabetical)":  (OrdinalEncoder(), OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1)),
    "one-hot":               (OneHotEncoder(handle_unknown="ignore", sparse_output=False),
                            OneHotEncoder(handle_unknown="ignore", min_frequency=20, sparse_output=False)),
    "target (cross-fitted)": (TargetEncoder(cv=KFold(5, shuffle=True, random_state=0)), TargetEncoder(cv=KFold(5, shuffle=True, random_state=0))),
}
rows = []
for name, (ce_, pe) in setups.items():
    for mname, model in [("logistic", LogisticRegression(max_iter=2000)), ("boosting", HistGradientBoostingClassifier(random_state=0))]:
        steps = [prep(ce_, pe)] + ([StandardScaler(with_mean=False)] if mname == "logistic" else []) + [model]
        p = make_pipeline(*steps).fit(train[X_cols], train["churn"])
        rows.append((name, mname, roc_auc_score(test["churn"], p.predict_proba(test[X_cols])[:, 1])))
res = pd.DataFrame(rows, columns=["encoding", "model", "AUC"]).pivot(index="encoding", columns="model", values="AUC")
print(res.round(3))

A Complete Encoding Pipeline 🔗पूरी Encoding Pipeline 🔗完整的编码管道 🔗

Put the right encoder on each group of columns with ColumnTransformer, chain it with the model in a Pipeline, and every category list, order and target mean is learned from training data only — even inside cross-validation.

ColumnTransformer से columns के हर समूह पर सही encoder लगाएँ, Pipeline में model से जोड़ें, और हर category list, क्रम और target mean सिर्फ़ training data से सीखा जाता है — cross-validation के अंदर भी।

用 ColumnTransformer 给每组列配上合适的编码器，再用 Pipeline 与模型串联，所有类别列表、顺序和目标均值都只从训练数据中学习 —— 即使在交叉验证内部也是如此。

# 8) A production pipeline: right encoder for each column type
import joblib
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.model_selection import cross_val_score
pre = ColumnTransformer([
    ("nominal", Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                          ("ohe", OneHotEncoder(handle_unknown="ignore", drop="if_binary"))]), ["city", "gender", "plan"]),
    ("ordinal", OrdinalEncoder(categories=order, handle_unknown="use_encoded_value", unknown_value=-1), ["income_bracket"]),
    ("high_card", TargetEncoder(cv=KFold(5, shuffle=True, random_state=0)), ["pincode"]),
    ("numeric", StandardScaler(), ["tenure_months"]),
])
pipe = Pipeline([("prep", pre), ("model", LogisticRegression(max_iter=2000))])
cols = ["city", "gender", "plan", "income_bracket", "pincode", "tenure_months"]
cv = cross_val_score(pipe, train[cols], train["churn"], cv=5, scoring="roc_auc")
pipe.fit(train[cols], train["churn"])
print(f"5-fold CV AUC {cv.mean():.3f} ± {cv.std():.3f} | test AUC {roc_auc_score(test['churn'], pipe.predict_proba(test[cols])[:, 1]):.3f}")
print("columns after encoding:", pipe["prep"].get_feature_names_out().size)
joblib.dump(pipe, "churn_pipeline.joblib")
new = pd.DataFrame([{"city": "Lucknow", "gender": "Male", "plan": "Basic",
                     "income_bracket": "Low", "pincode": "PIN999999", "tenure_months": 6}])
with warnings.catch_warnings(record=True) as w:                    # unknown city → all zeros, with a UserWarning
    warnings.simplefilter("always")
    p = joblib.load("churn_pipeline.joblib").predict_proba(new)[:, 1]
print(f"unseen city + pincode still predicts: {p[0]:.3f} | warnings raised: {len(w)} ({w[0].category.__name__})")

Make the Practice Data 🐍Practice Data बनाएँ 🐍生成练习数据 🐍

Run this first. City has an up-and-down effect in alphabetical order (so alphabetical codes carry almost no linear signal); income has a real order; pincode has 600 values with their own effects and a long tail of tiny ones; gender has no effect at all.

इसे पहले चलाएँ। City का असर वर्णमाला क्रम में ऊपर-नीचे है (इसलिए वर्णमाला codes में लगभग कोई linear संकेत नहीं); income में असली क्रम है; pincode के 600 मान हैं, सबका अपना असर, और छोटे pincodes की लंबी tail; gender का कोई असर नहीं।

请先运行它。城市的影响按字母顺序忽高忽低（所以字母编码几乎不含线性信号）；收入有真实顺序；邮编有 600 个取值，各有影响，并有大量样本极少的长尾；性别完全没有影响。

# 0) Practice data: customer churn with four kinds of categorical column
import numpy as np, pandas as pd
rng = np.random.default_rng(8)
n = 8000
cities = ["Bengaluru", "Chennai", "Delhi", "Hyderabad", "Jaipur", "Kolkata", "Mumbai", "Pune"]
city_eff = dict(zip(cities, [-0.7, 0.8, -0.6, 0.7, -0.5, 0.6, -0.4, 0.5]))   # up-down-up: no order at all
city = rng.choice(cities, n)
income = rng.choice(["Low", "Medium", "High", "Very High"], n, p=[.3, .35, .25, .1])
inc_eff = {"Low": 0.9, "Medium": 0.3, "High": -0.3, "Very High": -0.9}       # a real order
pins = np.array([f"PIN{400000 + k * 37}" for k in range(600)])
pin_w = 1 / np.arange(1, 601) ** 0.9; pin_w /= pin_w.sum()                    # a few big pincodes, many tiny ones
pin_eff = dict(zip(pins, rng.normal(0, 0.8, 600)))
pincode = rng.choice(pins, n, p=pin_w)
gender = rng.choice(["Female", "Male"], n)                                    # binary, no effect
plan = rng.choice(["Basic", "Plus", "Premium"], n, p=[.5, .35, .15])
tenure = rng.integers(1, 61, n)
z = (np.vectorize(city_eff.get)(city) + np.vectorize(inc_eff.get)(income) + np.vectorize(pin_eff.get)(pincode)
     - 0.025 * tenure - 0.3 + rng.normal(0, 0.3, n))
churn = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
pd.DataFrame({"city": city, "income_bracket": income, "pincode": pincode, "gender": gender, "plan": plan,
              "tenure_months": tenure, "churn": churn}).to_csv("customers.csv", index=False)
print(f"customers.csv: {n:,} rows, churn rate {churn.mean():.1%}")

Golden Rules of Categorical Encoding 🏆Categorical Encoding के सुनहरे नियम 🏆类别编码黄金法则 🏆