Scaling in Machine Learning

Why Feature Scaling Is Essential

Raw data arrives in the units humans find natural — age in years, salary in rupees, distance in kilometres, transaction count as a raw integer. Each feature inhabits its own numerical universe with its own range, its own spread, and its own magnitude. Machine learning algorithms do not know or care about units — they only see numbers. When one feature ranges from 0 to 1 and another ranges from 10,000 to 10,000,000, any algorithm that computes distances or optimises gradients will spend the vast majority of its mathematical energy on the large-scale feature — effectively ignoring the small-scale one entirely.

The Four Core Scaling Methods

Four scaling methods cover every scenario in data science. Each transforms feature values differently — and each is the correct answer for a different situation. Choosing the wrong one is as harmful as not scaling at all.

Min-Max Normalisation — Scaling to a Fixed Range

Min-Max Normalisation compresses every value into the range [0, 1] by subtracting the minimum and dividing by the range. The distribution shape is preserved exactly — the data looks the same, just rescaled. It is the standard choice for neural networks because bounded inputs produce stable gradient updates during backpropagation.

from sklearn.preprocessing import MinMaxScaler
import pandas as pd
import numpy  as np

# ── Manual calculation ────────────────────────────────
x_min = df['purchase_amount'].min()
x_max = df['purchase_amount'].max()
df['purchase_norm'] = (df['purchase_amount'] - x_min) / (x_max - x_min)

# ── sklearn MinMaxScaler ──────────────────────────────
scaler    = MinMaxScaler(feature_range=(0, 1))
num_cols  = ['age', 'purchase_amount', 'delivery_days', 'salary']

# CRITICAL: fit on train only — transform both
scaler.fit(X_train[num_cols])
X_train[num_cols] = scaler.transform(X_train[num_cols])
X_test[num_cols]  = scaler.transform(X_test[num_cols])

# ── Custom range for neural networks ─────────────────
scaler_neg = MinMaxScaler(feature_range=(-1, 1))
X_train[num_cols] = scaler_neg.fit_transform(X_train[num_cols])

# ── Check the output range ────────────────────────────
print(X_train[num_cols].min())   # should all be 0.0
print(X_train[num_cols].max())   # should all be 1.0

# ── Reverse transform to original values ──────────────
original_values = scaler.inverse_transform(X_train[num_cols])

Z-Score Standardisation — The Universal Default

Z-Score Standardisation transforms each feature to have mean = 0 and standard deviation = 1. It does not bound values to a fixed range — outliers remain proportionally distant from the mean, and negative values are perfectly valid. It is the default scaling choice for linear models, logistic regression, SVM, and PCA.

from sklearn.preprocessing import StandardScaler

# ── Manual Z-score ────────────────────────────────────
df['salary_std'] = (df['salary'] - df['salary'].mean()) / df['salary'].std()

# ── sklearn StandardScaler ────────────────────────────
scaler = StandardScaler()
num_cols = ['age', 'salary', 'credit_score', 'employment_years']

scaler.fit(X_train[num_cols])           # learn mean and std from train only
X_train[num_cols] = scaler.transform(X_train[num_cols])
X_test[num_cols]  = scaler.transform(X_test[num_cols])

# ── Verify: mean≈0, std≈1 ────────────────────────────
stats = pd.DataFrame(X_train[num_cols]).agg(['mean', 'std']).round(4)
print(stats)
# Expected:           age    salary  credit_score  employment_years
# mean             0.0000    0.0000        0.0000            0.0000
# std              1.0000    1.0000        1.0000            1.0000

# ── Access fitted parameters ──────────────────────────
print("Means:", dict(zip(num_cols, scaler.mean_.round(2))))
print("Stds :", dict(zip(num_cols, np.sqrt(scaler.var_).round(2))))

Robust Scaling — Resistant to Outliers

RobustScaler centres each feature on its median and scales by the interquartile range (IQR). Because median and IQR are insensitive to extreme values, a single outlier cannot distort the scaling of the entire dataset. This makes it the right choice when your data contains legitimate extreme values that you are not removing.

from sklearn.preprocessing import RobustScaler

# ── Default: uses IQR (Q1=25%, Q3=75%) ───────────────
scaler = RobustScaler(quantile_range=(25.0, 75.0))

scaler.fit(X_train[num_cols])
X_train[num_cols] = scaler.transform(X_train[num_cols])
X_test[num_cols]  = scaler.transform(X_test[num_cols])

# ── Tighter range for heavily contaminated data ───────
scaler_tight = RobustScaler(quantile_range=(10.0, 90.0))

# ── Manual RobustScaler (understand the math) ─────────
Q1 = df['claim_amount'].quantile(0.25)
Q3 = df['claim_amount'].quantile(0.75)
Q2 = df['claim_amount'].median()
IQR = Q3 - Q1
df['claim_robust'] = (df['claim_amount'] - Q2) / IQR

# ── Compare: what happens to outliers after scaling ───
outlier_val = 4200000
std_scaled   = (outlier_val - df['claim_amount'].mean()) / df['claim_amount'].std()
rob_scaled   = (outlier_val - Q2) / IQR
print(f"StandardScaler: {std_scaled:.2f}")  # ≈ 6.3 (still large but less distorting)
print(f"RobustScaler  : {rob_scaled:.2f}")  # ≈ 68.6 (outlier preserved as extreme)

Max-Abs Scaling — For Sparse Data

MaxAbsScaler divides each value by the maximum absolute value of that feature, scaling to [−1, +1]. Crucially, it does not centre the data — it does not subtract the mean. This means zero values stay zero, making it ideal for sparse matrices like TF-IDF text features or count matrices where most values are zero and shifting would destroy sparsity.

from sklearn.preprocessing import MaxAbsScaler
from sklearn.feature_extraction.text import TfidfVectorizer

# ── TF-IDF → MaxAbsScaler pipeline ───────────────────
vectorizer = TfidfVectorizer(max_features=5000)
X_sparse   = vectorizer.fit_transform(df['review_text'])

scaler_mas = MaxAbsScaler()
X_scaled   = scaler_mas.fit_transform(X_sparse)  # preserves sparsity

# ── For numeric data with both negative and positive ──
scaler_mas.fit(X_train[num_cols])
X_train[num_cols] = scaler_mas.transform(X_train[num_cols])

# Check: values in [-1, +1]
print(X_train[num_cols].abs().max())    # should all be 1.0

Algorithm Scaling Requirements — The Definitive Guide

Not all algorithms are sensitive to feature scale. The fundamental question is: does the algorithm compute distances, optimise gradients, or apply regularisation penalties? If yes, scaling is essential. If the algorithm makes decisions based on feature thresholds (trees), scaling has no effect.

The Data Leakage Trap — The Most Dangerous Scaling Mistake

The single most common and most damaging mistake in feature scaling is fitting the scaler on the full dataset before splitting into train and test sets. This causes data leakage — test set statistics (minimum, maximum, mean, standard deviation) contaminate the scaler's parameters, and your validation metrics become optimistically biased estimates of real-world performance.

# ── ❌ WRONG: Fit on full dataset (data leakage) ──────
scaler.fit_transform(df[num_cols])       # leaks test set info into scaler
X_train, X_test = train_test_split(df)  # too late — damage done

# ── ✅ CORRECT: Fit on train only ─────────────────────
X_train, X_test = train_test_split(df, test_size=0.2, random_state=42)
scaler.fit(X_train[num_cols])              # learn from train only
X_train[num_cols] = scaler.transform(X_train[num_cols])
X_test[num_cols]  = scaler.transform(X_test[num_cols])

# ── ✅ BEST: Pipeline (leakage-proof by design) ───────
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('model',  LogisticRegression(max_iter=1000))
])
pipe.fit(X_train, y_train)    # scaler.fit() called only on X_train
pipe.score(X_test, y_test)    # scaler.transform() called on X_test

Choosing the Right Scaler — Decision Guide

Complete Feature Scaling Pipeline

from sklearn.pipeline        import Pipeline
from sklearn.compose         import ColumnTransformer
from sklearn.preprocessing   import StandardScaler, MinMaxScaler, RobustScaler
from sklearn.preprocessing   import OneHotEncoder
from sklearn.impute           import SimpleImputer
from sklearn.ensemble         import GradientBoostingClassifier
import joblib

# ── Column definitions ────────────────────────────────
normal_num = ['age', 'credit_score']              # well-behaved numeric
robust_num = ['salary', 'claim_amount']           # has legitimate outliers
bounded    = ['pixel_intensity', 'confidence']   # needs [0,1] range
cat_cols   = ['city', 'gender', 'product_type']

# ── Sub-pipelines ─────────────────────────────────────
std_pipe = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler',  StandardScaler())
])

robust_pipe = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler',  RobustScaler())
])

minmax_pipe = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler',  MinMaxScaler())
])

cat_pipe = Pipeline([
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('ohe',     OneHotEncoder(handle_unknown='ignore', sparse_output=False))
])

# ── Column transformer ────────────────────────────────
preprocessor = ColumnTransformer([
    ('standard', std_pipe,    normal_num),
    ('robust',   robust_pipe, robust_num),
    ('minmax',   minmax_pipe, bounded),
    ('nominal',  cat_pipe,    cat_cols),
])

# ── Full pipeline with model ──────────────────────────
pipeline = Pipeline([
    ('prep',  preprocessor),
    ('model', GradientBoostingClassifier(n_estimators=200, random_state=42))
])

# ── Fit, evaluate, save ───────────────────────────────
pipeline.fit(X_train, y_train)
print(f"Test accuracy: {pipeline.score(X_test, y_test):.3f}")
joblib.dump(pipeline, 'production_pipeline.pkl')

# ── Load and predict on new raw data ─────────────────
pipeline_loaded = joblib.load('production_pipeline.pkl')
predictions = pipeline_loaded.predict(new_raw_data)  # auto-scales internally

Golden Rules of Feature Scaling