Feature Engineering & Feature Scaling

Why Feature Engineering Is the Most Impactful Step

Machine learning algorithms are mathematical functions — they can only learn from the signals you give them. If the raw data expresses those signals poorly, no amount of hyperparameter tuning or model complexity will recover them. Feature engineering is the art and science of transforming raw data into representations that expose the underlying patterns more clearly — creating new variables, combining existing ones, and scaling everything so that each feature speaks to the model in a language it can learn from.

Six Types of Feature Engineering

Interaction Features — Combining Columns

Two features that are individually weak can become a powerful predictor when combined. Interaction features capture the joint effect of two variables — a signal that exists only in their relationship, not in either variable alone.

import pandas as pd
import numpy  as np

# ── Ratio features ────────────────────────────────────
df['debt_to_income']       = df['total_debt'] / (df['annual_income'] + 1)
df['spend_per_delivery_day'] = df['purchase_amount'] / (df['delivery_days'] + 1)
df['revenue_per_visit']     = df['total_revenue'] / (df['visit_count'] + 1)

# ── Product features ──────────────────────────────────
df['age_x_income']     = df['age'] * df['income_bracket_enc']
df['price_x_quantity']  = df['unit_price'] * df['quantity']

# ── Difference features ───────────────────────────────
df['balance_change']    = df['balance_end'] - df['balance_start']
df['price_discount']    = df['original_price'] - df['sale_price']
df['discount_pct']      = df['price_discount'] / (df['original_price'] + 1) * 100

# ── Flag / binary interaction ─────────────────────────
df['is_high_value_weekend'] = (
    (df['purchase_amount'] > df['purchase_amount'].quantile(0.75)) &
    (df['is_weekend'] == True)
).astype(int)

# ── Always protect against division by zero ───────────
df['safe_ratio'] = np.where(df['denominator'] > 0,
                            df['numerator'] / df['denominator'],
                            0)

Polynomial Features — Capturing Non-Linear Relationships

A linear model can only fit straight lines. When the true relationship between a feature and the target is curved — diminishing returns, exponential growth, a U-shape — a linear model will never capture it regardless of how much data you have. Polynomial features solve this by creating x², x³, and cross-terms, allowing a linear model to fit non-linear patterns without switching to a non-linear algorithm.

from sklearn.preprocessing import PolynomialFeatures

# ── Degree 2: adds x², y², and x×y ───────────────────
poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X_train[['age', 'income']])

feature_names = poly.get_feature_names_out(['age', 'income'])
print(feature_names)
# ['age', 'income', 'age^2', 'age income', 'income^2']

# ── Manual polynomial (more readable) ────────────────
df['age_sq']      = df['age'] ** 2
df['age_cubed']   = df['age'] ** 3
df['income_sq']   = df['income'] ** 2
df['age_x_inc']   = df['age'] * df['income']    # cross-term

# ── IMPORTANT: Scale before polynomial! ──────────────
# Polynomial of unscaled features → huge numbers → numerical instability
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model  import Ridge

poly_pipeline = Pipeline([
    ('scaler', StandardScaler()),           # scale FIRST
    ('poly',   PolynomialFeatures(degree=2, include_bias=False)),
    ('model',  Ridge(alpha=1.0))
])
poly_pipeline.fit(X_train, y_train)

Date & Time Feature Extraction

Datetime columns are one of the richest sources of feature engineering in business data. A single timestamp contains many potential signals — hour of day, day of week, seasonality, recency, and cyclical patterns. Extracting these signals explicitly is far more informative than feeding a raw timestamp or integer to a model.

# ── Parse and extract all datetime features ───────────
df['order_date'] = pd.to_datetime(df['order_date'])

# Basic calendar features
df['year']        = df['order_date'].dt.year
df['month']       = df['order_date'].dt.month
df['day']         = df['order_date'].dt.day
df['hour']        = df['order_date'].dt.hour
df['day_of_week']  = df['order_date'].dt.dayofweek   # 0=Mon, 6=Sun
df['quarter']     = df['order_date'].dt.quarter
df['week_of_year'] = df['order_date'].dt.isocalendar().week

# Derived flag features
df['is_weekend']   = (df['day_of_week'] >= 5).astype(int)
df['is_friday']    = (df['day_of_week'] == 4).astype(int)
df['is_month_end']  = df['order_date'].dt.is_month_end.astype(int)
df['is_quarter_end']= df['order_date'].dt.is_quarter_end.astype(int)

# Recency feature: days since a reference date
ref_date = pd.Timestamp('2024-01-01')
df['days_since_signup']  = (df['order_date'] - df['signup_date']).dt.days
df['days_since_purchase'] = (pd.Timestamp.today() - df['order_date']).dt.days

# ── Cyclical encoding for hour and month ──────────────
# Prevents model from thinking Hour 23 is far from Hour 0
df['hour_sin']   = np.sin(2 * np.pi * df['hour']  / 24)
df['hour_cos']   = np.cos(2 * np.pi * df['hour']  / 24)
df['month_sin']  = np.sin(2 * np.pi * df['month'] / 12)
df['month_cos']  = np.cos(2 * np.pi * df['month'] / 12)

Binning & Discretisation — Grouping Continuous Values

Binning converts a continuous numeric column into ordered categorical buckets. This is useful when: the relationship between a feature and the target is step-wise rather than linear; you want to create interaction features between two continuous variables; or you want to let a linear model learn non-linear thresholds without using polynomial features.

# ── Equal-width bins (pd.cut) ─────────────────────────
# Each bin covers the same numeric range
df['age_group'] = pd.cut(
    df['age'],
    bins=[0, 25, 35, 45, 55, 100],
    labels=['18-25', '26-35', '36-45', '46-55', '55+'],
    right=True
)

# ── Equal-frequency bins (pd.qcut) ────────────────────
# Each bin contains the same number of rows
df['income_quartile'] = pd.qcut(
    df['annual_income'],
    q=4,
    labels=['Q1_Low', 'Q2_Med', 'Q3_High', 'Q4_VHigh']
)

# ── KBinsDiscretizer (sklearn — pipeline-friendly) ────
from sklearn.preprocessing import KBinsDiscretizer
kbd = KBinsDiscretizer(n_bins=5, encode='ordinal', strategy='quantile')
df['income_bin'] = kbd.fit_transform(df[['annual_income']])

# ── Custom domain bins ────────────────────────────────
bmi_bins   = [0, 18.5, 24.9, 29.9, 100]
bmi_labels = ['Underweight', 'Normal', 'Overweight', 'Obese']
df['bmi_category'] = pd.cut(df['bmi'], bins=bmi_bins, labels=bmi_labels)

# ── Encode binned features for modelling ──────────────
df['age_group_enc'] = df['age_group'].cat.codes     # ordinal
df = pd.get_dummies(df, columns=['age_group'], dtype=int) # one-hot

Aggregation Features — Capturing Group Behaviour

Aggregation features compute statistics about a group that a single row cannot express. A customer's individual purchase amount is limited information. The customer's average purchase amount over 12 months, their maximum single purchase, and the standard deviation of their purchases — these reveal spending personality. Aggregation features are among the most powerful in time-series and customer analytics problems.

# ── Customer-level aggregations ───────────────────────
cust_agg = df.groupby('customer_id').agg(
    cust_total_spend    = ('purchase_amount', 'sum'),
    cust_avg_spend      = ('purchase_amount', 'mean'),
    cust_max_spend      = ('purchase_amount', 'max'),
    cust_std_spend      = ('purchase_amount', 'std'),
    cust_order_count    = ('order_id',        'count'),
    cust_days_active    = ('order_date',      'nunique'),
    cust_avg_rating     = ('rating',          'mean'),
    cust_return_rate    = ('is_returned',     'mean'),
).reset_index()

# Merge back to original dataframe
df = df.merge(cust_agg, on='customer_id', how='left')

# ── Product-level aggregations ────────────────────────
prod_agg = df.groupby('product_id').agg(
    prod_avg_rating  = ('rating',       'mean'),
    prod_sale_count  = ('order_id',     'count'),
    prod_return_rate = ('is_returned',  'mean'),
    prod_avg_price   = ('unit_price',   'mean')
).reset_index()

# ── City-level aggregations ───────────────────────────
city_agg = df.groupby('city')['purchase_amount'].agg(
    city_avg_spend='mean', city_spend_std='std'
).reset_index()
df = df.merge(city_agg, on='city', how='left')

# ── Rolling window aggregations (time series) ─────────
df = df.sort_values(['customer_id', 'order_date'])
df['rolling_30d_spend'] = (df
    .groupby('customer_id')['purchase_amount']
    .transform(lambda x: x.rolling(30, min_periods=1).mean())
)

Domain-Specific Feature Engineering

The most powerful features are those derived from domain knowledge — features that encode what a human expert would consider when making the same prediction. These cannot be discovered by any automated feature engineering tool because they require understanding of the business context, not just the data.

Feature Scaling — Why It Must Come After Engineering

Feature scaling transforms numeric values into a consistent range so that distance-based and gradient-based algorithms treat all features equally. It must come after feature engineering because: (1) scaling before creating interaction features changes the meaning of the interaction; (2) newly created features may have very different ranges and also need scaling; and (3) binned and encoded features must be created from the raw values before any scaling is applied to numeric columns.

Scaling the Engineered Features

from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler

# All columns to scale — including engineered ones
scale_cols = [
    'age', 'income', 'salary',            # original numeric
    'debt_to_income', 'spend_per_day',       # engineered ratios
    'age_sq', 'income_sq', 'age_x_income',   # polynomial features
    'cust_avg_spend', 'cust_return_rate',    # aggregation features
    'days_since_signup', 'hour_sin',          # time features
]

# Standard: for linear models, SVM, PCA
std_scaler = StandardScaler()
std_scaler.fit(X_train[scale_cols])
X_train[scale_cols] = std_scaler.transform(X_train[scale_cols])
X_test[scale_cols]  = std_scaler.transform(X_test[scale_cols])

# Robust: for features with outliers (e.g. aggregated revenue)
robust_cols = ['cust_total_spend', 'cust_max_spend']
rob_scaler  = RobustScaler()
rob_scaler.fit(X_train[robust_cols])
X_train[robust_cols] = rob_scaler.transform(X_train[robust_cols])
X_test[robust_cols]  = rob_scaler.transform(X_test[robust_cols])

Complete Feature Engineering + Scaling Pipeline

In production, all feature engineering and scaling steps must be encapsulated in a single sklearn pipeline so they are applied identically at training and inference time. A custom transformer wraps the engineering logic; ColumnTransformer applies different scalers to different column groups.

from sklearn.pipeline        import Pipeline
from sklearn.compose         import ColumnTransformer
from sklearn.preprocessing   import StandardScaler, RobustScaler, MinMaxScaler
from sklearn.preprocessing   import OneHotEncoder, PolynomialFeatures
from sklearn.base             import BaseEstimator, TransformerMixin
from sklearn.impute           import SimpleImputer
from sklearn.ensemble         import GradientBoostingClassifier
import joblib

# ── 1. Custom feature engineering transformer ─────────
class FeatureEngineer(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None): return self
    def transform(self, X):
        X = X.copy()
        # Interaction features
        X['debt_to_income']  = X['debt']  / (X['income']  + 1)
        X['spend_per_day']   = X['spend']  / (X['days']    + 1)
        X['discount_pct']    = (X['orig_price'] - X['sale_price']) / (X['orig_price'] + 1)
        # Date features
        X['day_of_week']     = pd.to_datetime(X['order_date']).dt.dayofweek
        X['is_weekend']      = (X['day_of_week'] >= 5).astype(int)
        X['hour_sin']        = np.sin(2*np.pi*pd.to_datetime(X['order_date']).dt.hour/24)
        X['hour_cos']        = np.cos(2*np.pi*pd.to_datetime(X['order_date']).dt.hour/24)
        X['days_since_signup'] = (pd.Timestamp.today() - pd.to_datetime(X['signup_date'])).dt.days
        return X.drop(columns=['order_date', 'signup_date'])

# ── 2. Column groups after engineering ────────────────
standard_cols = ['age', 'income', 'credit_score', 'debt_to_income',
                 'spend_per_day', 'hour_sin', 'hour_cos', 'days_since_signup']
robust_cols   = ['total_spend', 'max_spend']    # outlier-prone
bounded_cols  = ['discount_pct', 'utilisation_pct']  # naturally [0,100]
cat_cols      = ['city', 'gender', 'product_type']

# ── 3. Preprocessing sub-pipelines ───────────────────
std_pipe    = Pipeline([('imp',SimpleImputer(strategy='median')),    ('sc',StandardScaler())])
robust_pipe = Pipeline([('imp',SimpleImputer(strategy='median')),    ('sc',RobustScaler())])
mm_pipe     = Pipeline([('imp',SimpleImputer(strategy='median')),    ('sc',MinMaxScaler())])
cat_pipe    = Pipeline([('imp',SimpleImputer(strategy='most_frequent')),
                         ('ohe',OneHotEncoder(handle_unknown='ignore',sparse_output=False))])

preprocessor = ColumnTransformer([
    ('std',    std_pipe,    standard_cols),
    ('robust', robust_pipe, robust_cols),
    ('mm',     mm_pipe,     bounded_cols),
    ('cat',    cat_pipe,    cat_cols),
])

# ── 4. Full pipeline ─────────────────────────────────
full_pipeline = Pipeline([
    ('engineer',    FeatureEngineer()),
    ('preprocessor',preprocessor),
    ('model',       GradientBoostingClassifier(n_estimators=300, random_state=42))
])

full_pipeline.fit(X_train, y_train)
print(f"Test AUC: {full_pipeline.score(X_test, y_test):.3f}")
joblib.dump(full_pipeline, 'feature_eng_pipeline.pkl')

# At inference — raw data in, predictions out
pipeline = joblib.load('feature_eng_pipeline.pkl')
predictions = pipeline.predict(new_raw_df)

Multicollinearity — Where It Enters & How to Fix It

Multicollinearity occurs when two or more features are so highly correlated with each other that one can be predicted from the others. It does not affect every algorithm — but for linear models, logistic regression, Ridge, Lasso, and KNN it silently destroys coefficient interpretability, inflates standard errors, and makes the model unstable. Crucially, it enters your dataset most often as a result of feature engineering itself — the very step designed to improve your model.

Which Models Are Harmed vs Safe

Detecting Multicollinearity

import pandas as pd
import numpy  as np
from statsmodels.stats.outliers_influence import variance_inflation_factor

# ── Method 1: Correlation matrix ─────────────────────
corr = X_train.corr().abs()
upper = corr.where(
    np.triu(np.ones(corr.shape, dtype=bool), k=1)
)
problem_pairs = [
    (col, row, upper.loc[row, col])
    for col in upper.columns
    for row in upper.index
    if upper.loc[row, col] > 0.85
]
print(pd.DataFrame(problem_pairs, columns=['feature_1','feature_2','correlation']))

# ── Method 2: Variance Inflation Factor (VIF) ────────
# The definitive test — VIF > 10 = serious problem
vif_data = pd.DataFrame({
    'feature': X_train.columns,
    'VIF':     [variance_inflation_factor(X_train.values, i)
               for i in range(X_train.shape[1])]
}).sort_values('VIF', ascending=False)

print(vif_data)
# VIF > 10  → serious multicollinearity — must fix
# VIF 5-10  → moderate — worth investigating
# VIF < 5   → acceptable

# ── Flag the worst offenders ──────────────────────────
high_vif = vif_data[vif_data['VIF'] > 10]
print(f"Features with VIF > 10: {high_vif['feature'].tolist()}")

Fixing Multicollinearity — Four Strategies

# ── Strategy 1: Drop one of each correlated pair ─────
# Keep the one with higher target correlation
target_corr = X_train.corrwith(y_train).abs()

def drop_multicollinear(X, threshold=0.85):
    corr = X.corr().abs()
    upper = corr.where(np.triu(np.ones(corr.shape, dtype=bool), k=1))
    to_drop = set()
    for col in upper.columns:
        correlated = upper.index[upper[col] > threshold].tolist()
        for row in correlated:
            # Drop whichever has lower target correlation
            drop = col if target_corr.get(col, 0) < target_corr.get(row, 0) else row
            to_drop.add(drop)
    print(f"Dropping {len(to_drop)} collinear features: {to_drop}")
    return X.drop(columns=list(to_drop))

X_train = drop_multicollinear(X_train, threshold=0.85)

# ── Strategy 2: Ridge Regression (L2 regularisation) ─
# Does NOT remove features — stabilises unstable coefficients
from sklearn.linear_model import RidgeCV
ridge = RidgeCV(alphas=[0.01, 0.1, 1.0, 10.0, 100.0], cv=5)
ridge.fit(X_train_scaled, y_train)
print(f"Best alpha: {ridge.alpha_}")

# ── Strategy 3: PCA — compress correlated features ───
from sklearn.decomposition import PCA
# All correlated features → a few orthogonal (uncorrelated) components
pca = PCA(n_components=0.95)   # keep 95% of variance
X_pca = pca.fit_transform(X_train_scaled)
print(f"Reduced from {X_train.shape[1]} to {X_pca.shape[1]} components")

# ── Strategy 4: Fix dummy variable trap ──────────────
# Always use drop_first=True for linear/logistic regression
df_encoded = pd.get_dummies(df, columns=['city', 'gender'],
                             drop_first=True, dtype=int)
# sklearn OneHotEncoder
from sklearn.preprocessing import OneHotEncoder
ohe = OneHotEncoder(drop='first', sparse_output=False)

Golden Rules of Feature Engineering & Scaling