Hypothesis Testing, Null vs Alternative Hypothesis,

The Courtroom, The Medicine & The Factory

Imagine you are a judge. A person stands before you accused of a crime. In every democratic legal system, the law starts from one powerful assumption: the accused is innocent until proven guilty. The burden of proof lies entirely with the prosecution. Unless the evidence is so overwhelming that it cannot plausibly be explained by chance or coincidence, the verdict must be not guilty.

Notice the judge does not prove innocence. The judge simply asks: "Is the evidence against this person strong enough to reject the assumption of innocence?" This is exactly how hypothesis testing works in statistics — and it is not a coincidence that statisticians borrowed the language of law to build it.

Hypothesis testing is the formal statistical procedure for deciding whether observed data provides sufficient evidence to reject a specific claim about a population. It is the engine behind every clinical trial, quality control system, A/B test, and scientific publication on the planet.

What Is a Hypothesis Test?

A statistical hypothesis is a testable claim about a population parameter — a mean, a proportion, a variance, a difference. The test uses sample data to decide how plausible that claim is. The entire procedure rests on one central question:

Null vs Alternative Hypothesis — In Depth

The null and alternative hypotheses are two competing statements about the world. Together they must be mutually exclusive and exhaustive. Understanding how to formulate them correctly is the most critical skill in hypothesis testing.

The Three Test Types

Significance Level α and the p-value

Before collecting any data you must set the significance level α — the maximum probability of wrongly rejecting H₀ you are willing to tolerate. After collecting data you compute the p-value, then compare the two to make your decision.

Type I & Type II Errors — The Cost of Being Wrong

No matter how carefully you design a test, two types of error are always possible. Think of a COVID test analogy: a Type I error is a false positive (testing positive when you are healthy); a Type II error is a false negative (testing negative when you are actually infected).

The Universal 5-Step Framework

Every hypothesis test — regardless of which specific test you use — follows the same five-step procedure. Memorising this structure means you will never lose track of where you are or what a result actually means.

One-tailed vs Two-tailed Tests

The choice between one-tailed and two-tailed must be made before looking at the data. Switching after seeing results to get a smaller p-value is a form of data manipulation called p-hacking.

Python Implementation

One-sample t-test — Factory Bolt

import numpy as np
from scipy import stats

np.random.seed(7)

# 25 bolt weight measurements (grams)
bolt_weights = np.random.normal(loc=51.2, scale=2.8, size=25)

mu_0  = 50       # H₀: μ = 50 g
alpha = 0.05

# Two-tailed one-sample t-test
t_stat, p_value = stats.ttest_1samp(bolt_weights, popmean=mu_0)

x_bar = np.mean(bolt_weights)
s     = np.std(bolt_weights, ddof=1)
se    = s / np.sqrt(len(bolt_weights))
df    = len(bolt_weights) - 1
t_crit = stats.t.ppf(1 - alpha/2, df=df)

print(f"x̄ = {x_bar:.4f} g  |  s = {s:.4f} g  |  SE = {se:.4f}")
print(f"t_stat   = {t_stat:.4f}")
print(f"t_crit   = ±{t_crit:.4f}  (df={df}, α={alpha})")
print(f"p_value  = {p_value:.4f}")

if p_value < alpha:
    print(f"\np={p_value:.4f} < α={alpha} → REJECT H₀")
    print("Mean bolt weight differs significantly from 50 g")
else:
    print(f"\np={p_value:.4f} ≥ α={alpha} → FAIL TO REJECT H₀")

One-tailed Test — Drug Trial (Right-tailed)

import numpy as np
from scipy import stats

np.random.seed(42)

# Blood pressure reduction (mmHg) for 30 trial patients
# H₀: μ ≤ 10  |  H₁: μ > 10  (right-tailed)
bp_reduction = np.random.normal(loc=12.5, scale=4.0, size=30)
mu_0, alpha  = 10, 0.05

t_stat, p_two = stats.ttest_1samp(bp_reduction, popmean=mu_0)

# For right-tailed: halve p-value only when t_stat > 0
p_one_right = p_two / 2 if t_stat > 0 else 1 - (p_two / 2)

print(f"Mean reduction: {np.mean(bp_reduction):.2f} mmHg")
print(f"t-statistic:    {t_stat:.4f}")
print(f"p (one-tailed): {p_one_right:.4f}")

if p_one_right < alpha:
    print(f"\nREJECT H₀ — new drug produces significantly greater reduction")
else:
    print(f"\nFAIL TO REJECT H₀")

Two-sample Welch's t-test — A/B Test

import numpy as np
from scipy import stats

np.random.seed(0)

# Conversion data for two button designs
# H₀: μ_A = μ_B  |  H₁: μ_A ≠ μ_B  (two-tailed)
conversions_A = np.random.binomial(1, 0.042, 500)
conversions_B = np.random.binomial(1, 0.047, 500)
alpha = 0.05

# Welch's t-test — always use equal_var=False in practice
t_stat, p_value = stats.ttest_ind(conversions_A, conversions_B,
                                   equal_var=False)

print(f"Button A rate: {conversions_A.mean()*100:.2f}%")
print(f"Button B rate: {conversions_B.mean()*100:.2f}%")
print(f"t-statistic:   {t_stat:.4f}")
print(f"p-value:       {p_value:.4f}")

if p_value < alpha:
    print(f"\nREJECT H₀ — significant difference between buttons")
else:
    print(f"\nFAIL TO REJECT H₀ — no significant difference detected")

Computing Cohen's d — Effect Size

import numpy as np
from scipy import stats

def cohen_d(group1, group2):
    """Pooled Cohen's d for two independent groups."""
    n1, n2 = len(group1), len(group2)
    s1, s2 = np.std(group1, ddof=1), np.std(group2, ddof=1)
    pooled_std = np.sqrt(((n1-1)*s1**2 + (n2-1)*s2**2) / (n1+n2-2))
    return (np.mean(group1) - np.mean(group2)) / pooled_std

np.random.seed(42)
control = np.random.normal(10, 4, 30)
treated = np.random.normal(12.5, 4, 30)

t_stat, p_val = stats.ttest_ind(treated, control, equal_var=False)
d = cohen_d(treated, control)

print(f"t = {t_stat:.4f},  p = {p_val:.4f}")
print(f"Cohen's d = {d:.4f}")

# Interpret effect size
if   abs(d) < 0.2: label = "negligible"
elif abs(d) < 0.5: label = "small"
elif abs(d) < 0.8: label = "medium"
else:               label = "large"
print(f"Effect size: {label}")

# Crucial lesson: statistically significant doesn't mean practically important.
# Always report d alongside p-value.

Choosing the Right Test — Quick Reference

Golden Rules