Data Splitting Mastery

Why Data Splitting Matters

Every machine learning model has one job: generalise well to data it has never seen before. If you train and evaluate on the same data, your accuracy number is a lie — the model has simply memorised the answers. Data splitting is the practice of partitioning your dataset into separate subsets so that training, tuning, and final evaluation each happen on independent data.

The Three-Set Framework

The Workflow Order

Train-Test Split (The Holdout Method)

The simplest form of data splitting is the holdout method: randomly partition the dataset once into a training set and a test set. It is fast, easy to understand, and the right default when you have a large dataset (typically >10,000 rows).

How It Works — Shuffle Then Split

Python Implementation

from sklearn.model_selection import train_test_split
import pandas as pd

# Load dataset
df = pd.read_csv('house_prices.csv')
X  = df.drop('price', axis=1)
y  = df['price']

# Simple 80/20 split
X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.20,
    random_state=42,
    shuffle=True
)
print(f"Training rows : {len(X_train)}")
print(f"Test rows     : {len(X_test)}")

Key Parameters

Stratified Split — Preserving Class Balance

When classes are imbalanced, a random split may put all minority cases in training and none in the test set. stratify=y guarantees every split mirrors the original class ratio.

The Validation Set

Once you add hyperparameter tuning — choosing learning rate, tree depth, regularisation strength — you need a third partition: the validation set. This is the data you use to compare models and tune parameters without contaminating the test set.

Creating a Three-Way Split in Python

from sklearn.model_selection import train_test_split

# Step 1: split off the test set first (20%)
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.20, random_state=42, stratify=y
)

# Step 2: split remaining 80% → train (75%) + val (25%)
# 0.25 × 0.80 = 0.20 of total → gives 60/20/20 overall
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.25, random_state=42, stratify=y_temp
)

print(f"Train : {len(X_train)} ({len(X_train)/len(X)*100:.0f}%)")
print(f"Val   : {len(X_val)}   ({len(X_val)/len(X)*100:.0f}%)")
print(f"Test  : {len(X_test)}  ({len(X_test)/len(X)*100:.0f}%)")

K-Fold Cross-Validation

A single validation split is sensitive to which rows happen to land where. Cross-validation solves this by repeating the train/evaluate cycle multiple times on different partitions and averaging the results. Every data point gets to be in the validation set exactly once.

from sklearn.model_selection import KFold, cross_val_score
from sklearn.ensemble       import RandomForestClassifier
import numpy as np

model = RandomForestClassifier(n_estimators=100, random_state=42)
kf    = KFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_val_score(model, X_train, y_train, cv=kf, scoring='accuracy')
print("Fold scores :", np.round(scores, 4))
print(f"Mean        : {scores.mean():.4f}")
print(f"Std Dev     : {scores.std():.4f}")
print(f"95% CI      : {scores.mean():.4f} ± {2*scores.std():.4f}")

The Math Behind K-Fold Scoring

Stratified K-Fold — Preserving Class Balance

Regular K-Fold shuffles randomly. With imbalanced classes, a random fold could end up with zero minority cases in the validation set — making your accuracy score completely unreliable. StratifiedKFold ensures each fold maintains the same class proportions as the full dataset.

Leave-One-Out Cross-Validation (LOOCV)

LOOCV is the extreme case of k-fold where k = N. Each iteration trains on N−1 samples and validates on a single sample. Every data point gets one turn as the test case. It gives the least biased estimate but is computationally expensive for large datasets.

Time Series Cross-Validation

Standard k-fold must not be used for time-series data — using future data to predict the past is data leakage. TimeSeriesSplit enforces temporal order: training always uses past data, validation always uses future data.

from sklearn.model_selection import TimeSeriesSplit
from sklearn.linear_model   import Ridge

tscv     = TimeSeriesSplit(n_splits=5)
model_ts = Ridge(alpha=1.0)

scores = cross_val_score(
    model_ts, X_timeseries, y_timeseries,
    cv=tscv,
    scoring='neg_root_mean_squared_error'
)
rmse_scores = -scores
print(f"RMSE per fold : {np.round(rmse_scores, 2)}")
print(f"Mean RMSE     : {rmse_scores.mean():.3f}")

Which Technique Should You Use?

Nested Cross-Validation & Pipelines

When you use the same CV loop for both model selection and performance estimation, the estimate is optimistically biased. The solution is nested cross-validation — an outer loop for unbiased performance estimation, and an inner loop for hyperparameter search.

from sklearn.model_selection import GridSearchCV, cross_val_score, StratifiedKFold
from sklearn.svm            import SVC

param_grid = {'C': [0.1, 1, 10], 'kernel': ['rbf', 'linear']}
inner_cv   = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
outer_cv   = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

clf = GridSearchCV(SVC(), param_grid, cv=inner_cv, scoring='accuracy')

nested_scores = cross_val_score(clf, X, y, cv=outer_cv, scoring='accuracy')
print(f"Nested CV Accuracy: {nested_scores.mean():.4f} ± {nested_scores.std():.4f}")

Data Leakage — The #1 Mistake in ML

Data leakage happens when information from outside the training set leaks into the model during training, making it look better than it really is. It is the single most common reason ML models fail in production despite appearing excellent during development.

Three Types of Leakage

Leaky Pipeline vs Clean Pipeline

Using sklearn Pipeline to Prevent Leakage Automatically

from sklearn.pipeline     import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble      import GradientBoostingClassifier

# Pipeline ensures scaler is fit ONLY on training data in each fold
pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('model',  GradientBoostingClassifier(random_state=42))
])

# cross_val_score + Pipeline = leakage-free evaluation
scores = cross_val_score(
    pipe, X, y,
    cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=42),
    scoring='accuracy'
)
print(f"Pipeline CV: {scores.mean():.4f} ± {scores.std():.4f}")

Summary & Golden Rules