What Is Train-Test Split?
- 1 hour ago
- 28 min read
A model can score 99% on the data it was trained on and still fail the moment it meets a real customer, a real transaction, or a real patient. That gap between "looks great in training" and "falls apart in the real world" is exactly the problem a train-test split exists to catch, by holding back a portion of data the model never gets to study before grading its final exam.
TL;DR
A train-test split divides a dataset into a training set the model learns from and a test set used only to measure performance on unseen data.
The test set must stay untouched during training and tuning, or the resulting score stops reflecting real-world generalization.
The right splitting method—random, stratified, grouped, or chronological—depends on how your data is structured, not on habit.
Data leakage—letting test information influence training in any way—is one of the most common causes of misleadingly optimistic results.
A single split gives one estimate; cross-validation and a protected final test set work together, rather than replacing each other.
What is a train-test split?
A train-test split is a method for dividing a dataset into two parts: a training set, used to fit a machine learning model, and a test set, held back and used only afterward to evaluate how well the model performs on new, unseen data. This separation estimates real-world performance and helps detect overfitting.
Table of Contents
What a Train-Test Split Actually Means
In plain English, a train-test split takes one dataset and cuts it into two pieces. One piece—the training set—is shown to the model so it can learn patterns. The other piece—the test set—is locked away and shown to the model only after training is finished, purely to check how it performs.
More formally, given a dataset of (n) observations, a train-test split partitions the observations into a training subset and a test subset that do not overlap. The model's parameters are estimated using only the training subset. The test subset is then used, exactly once ideally, to compute an unbiased estimate of the model's performance on data drawn from the same underlying population but not seen during fitting.
The words practitioners use here matter. "Held-out" data is data deliberately withheld from training. "Unseen" data is data the model has never encountered in any form, including during preprocessing. "Out-of-sample" is a statistics term for data outside the sample used to fit a model—train-test splitting is a practical way of manufacturing out-of-sample data from a single dataset.
A simple, technically accurate analogy: think of the training set as the practice questions a student studies from, and the test set as the exam questions the student has never seen. A student who "aces" a test made entirely of practice questions they already memorized hasn't demonstrated understanding—only recall. The same logic applies to a machine learning model.
Importantly, the two subsets don't always come from randomly slicing one big pile of rows. They can also come from data that is naturally separated—by time period (last year's transactions vs. this year's), by user (one group of patients vs. another), by location (stores in one region vs. another), by device, or by collection process (data gathered before a system change vs. after). Recognizing which kind of separation applies to your problem is often more important than the exact split percentage you choose.
Why Train-Test Splitting Matters
Generalization is the real goal
The purpose of building a predictive model is almost never to perform well on the exact rows it was trained on—it's to perform well on new rows it hasn't seen yet. That ability is called generalization. A model that memorizes training data instead of learning transferable patterns has overfit; a model too simple to capture the real pattern has underfit. Both failure modes look fine if you only ever check performance on training data, which is precisely why training accuracy alone is not sufficient evidence that a model works.
In-sample vs. out-of-sample performance
In-sample performance measures how well a model fits the data it already saw. Out-of-sample performance measures how well it predicts new data. The two numbers can diverge sharply. A decision tree with no depth limit can reach close to 100% training accuracy on almost any dataset by essentially memorizing it, while its out-of-sample accuracy might be far lower. The bias-variance tradeoff explains why: increasing model flexibility reduces bias but increases variance, and variance is exactly the sensitivity to the specific training sample that hurts generalization.
A test score is an estimate, not a guarantee
Even a well-constructed test set only tells you how the model performed on that particular sample of unseen data. It is a statistical estimate of future performance, not a certification. Two things follow from this. First, the test distribution should resemble the distribution the model will actually meet in deployment—a test set of hospital records from one city may not predict performance in a different healthcare system. Second, any estimate based on a finite sample carries sampling error: the same model, evaluated on a different but equally valid test sample, would likely produce a slightly different number.
Why a single split can be noisy
For small datasets, this sampling error can be large. A test set of 30 examples can easily show 80% accuracy on one random split and 70% on another, purely from which specific examples happened to land in the test set—no change to the model at all. This is one reason practitioners lean on cross-validation for small or noisy datasets, a point covered later in this guide.
Table: In-sample vs. out-of-sample performance
Aspect | In-Sample (Training) | Out-of-Sample (Test) |
Data seen by model | Yes | No |
Reflects real-world use | Not reliably | Yes, approximately |
Sensitive to overfitting | Hides it | Reveals it |
Typical use | Fitting parameters | Final performance estimate |
Risk if misused | Inflated confidence | Biased if reused repeatedly |
How a Train-Test Split Works End to End
A reliable splitting workflow follows a consistent order. Skipping steps, or doing them out of order, is where most evaluation mistakes originate.
Define the prediction task and target population. What exactly is being predicted, for whom, and under what conditions will the model be used?
Inspect the dataset. Check for repeated entities, time ordering, duplicate or near-duplicate rows, label quality, and class imbalance before choosing anything.
Select an appropriate splitting strategy—random, stratified, grouped, or chronological—based on what step 2 revealed.
Split the data first, before fitting any learned preprocessing steps such as scalers or encoders.
Fit preprocessing only on the training data. Apply the already-fitted transformation to validation and test data; never re-fit on them.
Train the model using the transformed training data.
Use a validation set or cross-validation for model comparisons and hyperparameter decisions—not the test set.
Evaluate the finalized process on the untouched test set, ideally once.
Report the metric, the splitting strategy, the random seed (if applicable), and known limitations.
Monitor performance after deployment, since real-world data can drift away from the distribution the test set represented.
Steps 4 through 6 can be combined safely using scikit-learn's Pipeline, which chains preprocessing and the estimator together so that fit() is always called on training data alone and transform()/predict() is applied consistently to everything else. This single design choice eliminates a large share of leakage mistakes almost automatically, covered in more detail later.
Choosing a Train-Test Split Ratio
There is no split ratio that is correct for every dataset. Practitioners commonly see 50/50, 60/40, 70/30, 75/25, 80/20, and 90/10, but each one is a trade-off between two competing needs: enough training data for the model to learn well, and enough test data for the performance estimate to be trustworthy.
Table: Common split ratios and typical rationale
Ratio (train/test) | When it's often seen | Trade-off |
50/50 | Very large datasets, or when validation needs equal statistical power | Wastes training data if the dataset isn't huge |
60/40 | Exploratory work, small models | More robust test estimate, less training signal |
70/30 | General-purpose default for small-to-medium data | Balanced, commonly cited as a reasonable starting point |
80/20 | Very common default in tutorials and libraries | Good balance for medium datasets; can be unstable if data is small |
90/10 | Large datasets, deep learning | 10% of a huge dataset is still statistically ample |
The percentage alone can be misleading without considering the absolute count. Ten percent of one million rows is 100,000 test examples—more than enough for a precise estimate. Twenty percent of 100 rows is only 20 test examples, which is often too few to trust, especially if there are multiple classes.
Consider three quick arithmetic examples using (n_{\text{test}} \approx p \times n) and (n_{\text{train}} = n - n_{\text{test}}):
(n = 500), (p = 0.2): roughly 100 test rows, 400 training rows.
(n = 50{,}000), (p = 0.2): roughly 10,000 test rows, 40,000 training rows.
(n = 120), (p = 0.3): roughly 36 test rows, 84 training rows—likely too small for a stable estimate if there are several classes.
The choice should reflect: total sample size, model complexity, the number of classes, how rare the minority class is, how much statistical precision the evaluation needs, computational cost of retraining, whether records are naturally grouped, how much time coverage the data spans, the conditions the model will face in deployment, and whether cross-validation will also be used alongside the holdout split. When a dataset is small, favor a larger test set proportionally, or rely more on cross-validation than on a single fixed ratio. When a dataset is large, a smaller test percentage often still yields a large and precise test set while preserving more data for training.
Training, Validation, and Test Sets
These three terms get used loosely, but they serve distinct roles.
Training set: the data the model directly learns from—weights, coefficients, or split rules are fit here.
Validation set: a separate slice used during development to compare models, tune hyperparameters, and catch overfitting before touching the test set.
Test set: reserved for one final, unbiased estimate after every modeling decision has been made.
The validation set exists because model development involves dozens or hundreds of small decisions—which algorithm, which hyperparameters, which features. If every one of those decisions were checked against the test set, the test set would gradually become another validation set, and the resulting "final" score would be optimistic rather than honest. This phenomenon is sometimes called the test set "wearing out" through repeated human decision-making.
Two-way splitting (train/test only) is common for simple projects or when cross-validation substitutes for a dedicated validation set. Three-way splitting (train/validation/test) is common for projects involving hyperparameter search or model comparison. Cross-validation, covered in a later section, can reduce or eliminate the need for a separate fixed validation set by reusing the training data more efficiently across folds.
Table: Purpose and allowed uses
Set | Purpose | Allowed uses | Prohibited uses |
Training | Fit model parameters | Model fitting, feature learning | Final evaluation |
Validation | Guide development decisions | Hyperparameter tuning, model comparison, early stopping | Reporting as the final score |
Test | Estimate real-world performance | One (or very few) final evaluations | Any tuning or repeated peeking |
The ideal workflow settles the modeling approach using training and validation data (or cross-validation), and only then evaluates once on the test set, treating that number as the final answer for this iteration of the project.
Scikit-Learn's train_test_split() Explained
Scikit-learn's train_test_split is the standard tool for producing a random holdout split in Python. Its current documented signature is:
sklearn.model_selection.train_test_split(
*arrays,
test_size=None,
train_size=None,
random_state=None,
shuffle=True,
stratify=None
)A minimal, correct example:
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.2,
random_state=42
)Here is what each part does:
X, y: the feature matrix and target array (or any number of same-length arrays) to split. Every array passed in must have the same number of samples, since the function slices them together to keep rows aligned.
test_size=0.2: reserves 20% of the data for the test set. It can be a float between 0 and 1 (a proportion) or an integer (an absolute count of samples).
random_state=42: fixes the internal random number generator so the same split is produced every time the code runs. The number 42 has no statistical meaning—it's a conventional placeholder seed, not a "correct" value. Any fixed integer works identically well for reproducibility.
Return order: the function always returns the split arrays in the same relative order they were passed, doubled—X_train, X_test, y_train, y_test for two input arrays, following the pattern train, test for each input in sequence.
shuffle=True (default): rows are randomly shuffled before splitting. This is appropriate for independent, identically distributed data but inappropriate for ordered time-series data, where shuffling would leak future information into the training set.
stratify=None (default): when set to an array such as stratify=y, the split preserves the class proportions of y in both the train and test subsets, which is valuable for classification problems with class imbalance.
Regarding defaults: if neither test_size nor train_size is specified, test_size defaults to 0.25 according to current scikit-learn documentation—but this default only applies when both parameters are left unset; specifying either one overrides it. It is a good habit to set test_size explicitly rather than relying on this default, since being explicit makes the split ratio visible to anyone reading the code later.
Two clarifications worth internalizing. First, reproducibility and validity are different things. Setting random_state=42 guarantees you get the same split every time, but it does not make that split statistically sound—a reproducible bad split (say, one that happens to separate all of one class into the test set) is still a bad split. Second, scikit-learn's current documentation states that if shuffle=False, then stratify must be None—the two options are incompatible, because stratification requires randomly reassigning rows by class, which is meaningless if the data must stay in its original order (as with time series).
A Complete Classification Example
This example uses scikit-learn's built-in breast cancer dataset, a real, commonly used benchmark for binary classification, and follows a leakage-safe pattern using Pipeline.
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, roc_auc_score
data = load_breast_cancer()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(
X, y,
test_size=0.2,
random_state=42,
stratify=y
)
pipeline = Pipeline([
("scaler", StandardScaler()),
("model", LogisticRegression(max_iter=1000))
])
pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_test)
y_proba = pipeline.predict_proba(X_test)[:, 1]
print(classification_report(y_test, y_pred))
print("ROC AUC:", roc_auc_score(y_test, y_proba))Walking through the logic: stratify=y keeps the ratio of malignant to benign cases roughly equal in both the training and test subsets, which matters because the classes are not perfectly balanced. The Pipeline bundles StandardScaler and LogisticRegression together, so calling pipeline.fit(X_train, y_train) fits the scaler's mean and standard deviation using only training rows, then fits the model on the scaled training data—the test set never influences the scaler's parameters. classification_report prints precision, recall, and F1-score per class, while roc_auc_score reports a single ranking-quality metric.
Accuracy alone can be misleading here: if 90% of cases were benign, a model that always predicts "benign" would score 90% accuracy while catching zero malignant cases—a dangerous result in a medical context. Precision, recall, and ROC AUC surface that failure where raw accuracy would hide it. The exact numeric scores from this code will vary slightly depending on the scikit-learn version and hardware, so they are intentionally omitted here rather than presented as guaranteed output.
A Complete Regression Example
Regression predicts a continuous number rather than a category, so stratifying on the raw target the way you would for classification does not apply in the usual sense.
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np
data = fetch_california_housing()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(
X, y,
test_size=0.2,
random_state=42
)
pipeline = Pipeline([
("scaler", StandardScaler()),
("model", Ridge(alpha=1.0))
])
pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_test)
mae = mean_absolute_error(y_test, y_pred)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
r2 = r2_score(y_test, y_pred)
print(f"MAE: {mae:.3f}, RMSE: {rmse:.3f}, R²: {r2:.3f}")This example does not use stratify, because the target y here is a continuous house-value figure, not a set of discrete categories. MAE (mean absolute error) reports the average absolute size of the prediction error in the same units as the target—here, hundreds of thousands of dollars. RMSE (root mean squared error) penalizes large errors more heavily than MAE because errors are squared before averaging. R² describes the proportion of variance in the target explained by the model, on a scale where 1.0 is a perfect fit and 0 means the model performs no better than always predicting the average. All three numbers should be interpreted in the context of the target variable—an MAE of 0.5 means very different things for house prices measured in hundreds of thousands versus a target measured in single digits.
Stratified Splitting
Stratification means splitting the data so that each subset preserves, as closely as possible, the same proportion of each class found in the full dataset. It matters most for classification problems where classes are imbalanced, because a plain random split can, by chance, place disproportionately few examples of a rare class into the test set—or even none at all.
Consider a fraud-detection dataset with 980 legitimate transactions and 20 fraud cases (98%/2%). A random 80/20 split without stratification could easily place only 2 or 3 fraud cases in the 200-row test set, making any fraud-detection metric extremely noisy. With stratify=y, both the training and test sets would keep approximately the same 98%/2% ratio.
from sklearn.model_selection import train_test_split
from collections import Counter
# y has classes: 0 (legitimate) and 1 (fraud), imbalanced ~98/2
print("Before split:", Counter(y))
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
print("Train distribution:", Counter(y_train))
print("Test distribution:", Counter(y_test))Stratification does not solve severe imbalance by itself—it only ensures the imbalance is represented proportionally rather than distorted by random chance. If a class has only 5 total examples, even a stratified split may leave too few in the test set to draw a reliable conclusion, and the split may fail outright if a class has fewer members than the number of splits requested. Multilabel problems, where each row can belong to multiple classes simultaneously, need specialized multilabel-stratification approaches, since simple single-label stratification does not account for co-occurring label combinations. It's also worth distinguishing a stratified holdout split, which stratifies one single train/test division, from stratified cross-validation (StratifiedKFold), which stratifies each of several repeated folds.
Group-Aware Splitting
Rows in a dataset are not always independent of each other. Multiple images from the same patient, multiple transactions from the same customer, several sensor readings from the same machine, or many documents from the same author are all examples of grouped data. If a random split places some rows from a patient in the training set and other rows from that same patient in the test set, the model can effectively "recognize" the patient rather than learn a pattern that generalizes to new patients—producing an evaluation score that looks great but doesn't hold up on genuinely new individuals.
The fix is to split by group rather than by row, ensuring every row belonging to a given entity ends up entirely in one subset or the other. Scikit-learn provides GroupShuffleSplit for a single group-aware holdout split and GroupKFold for group-aware cross-validation; StratifiedGroupKFold combines grouping with class stratification when both constraints apply.
from sklearn.model_selection import GroupShuffleSplit
# groups: an array where each entry is, e.g., the patient ID for that row
splitter = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(splitter.split(X, y, groups=groups))
X_train, X_test = X[train_idx], X[test_idx]
y_train, y_test = y[train_idx], y[test_idx]Note that the requested test_size here refers to a proportion of groups, not necessarily a proportion of rows—if some groups contain many more rows than others, the resulting row-level split percentage may differ somewhat from the requested value.
Time-Series and Chronological Splitting
When observations are ordered in time and the goal is to predict the future from the past, random shuffling is usually the wrong approach. Shuffling would let information from later time points leak into the training set, letting the model implicitly "see the future" during training—a mistake known as look-ahead bias or temporal leakage. The realistic evaluation setup instead trains on the past and tests on the future, exactly mirroring how the model will be used once deployed.
Scikit-learn's TimeSeriesSplit implements this by producing successive train/test folds where the training set only ever contains observations earlier than the test fold.
from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5)
for fold, (train_idx, test_idx) in enumerate(tscv.split(X)):
X_train, X_test = X[train_idx], X[test_idx]
y_train, y_test = y[train_idx], y[test_idx]
# fit and evaluate the model per fold hereTwo evaluation styles are common: an expanding window, where each successive training fold keeps growing to include all prior data, and a rolling window, where the training fold has a fixed size and slides forward in time, discarding the oldest observations. Some workflows also insert a gap or "embargo" period between training and test folds to avoid subtle leakage when features are computed using rolling calculations that span the boundary. Practitioners should also check whether the training data spans enough seasons or cycles to capture seasonality, and remain aware that a model's accuracy can degrade over time due to distribution drift as real-world conditions shift.
Not every dataset with a timestamp automatically needs a chronological split—the deciding factor is the actual prediction scenario. A dataset with timestamps used only to describe when a static event occurred, with no notion of predicting forward in time, might not require temporal ordering at all. The right question is always: "In deployment, will the model only ever see information from before the moment of prediction?" If yes, the evaluation split should mirror that.
Data Leakage
Data leakage happens whenever information that would not be available at genuine prediction time ends up influencing the model or its evaluation, producing performance numbers that look better than what the model will achieve in the real world. It is not the same thing as ordinary overfitting—overfitting comes from a model that is too flexible for the amount of clean data available; leakage comes from the evaluation setup itself being compromised, regardless of how the model was built.
Leakage takes several distinct forms:
Target leakage: a feature that is itself derived from the outcome, or only knowable after the outcome occurs.
Train-test contamination: any overlap of information between the training and test partitions.
Preprocessing leakage: fitting a scaler, encoder, or imputer on the full dataset before splitting.
Duplicate or near-duplicate leakage: identical or nearly identical rows appearing in both splits.
Group leakage: records from the same entity appearing in both training and test sets, discussed above.
Temporal leakage: future information reaching the training set.
Feature-selection leakage: choosing which features to keep based on statistics computed from the full dataset, including the test portion.
Imputation leakage: filling missing values using statistics (like the overall mean) computed across the whole dataset.
Scaling leakage: fitting a scaler's parameters on all the data rather than training data alone.
Encoding leakage: especially target encoding, where a categorical value is replaced by a statistic of the target—if computed using test rows, that's leakage.
Oversampling or synthetic-sampling leakage: generating synthetic minority-class examples before splitting, so synthetic copies derived from test rows end up in training.
Test-set tuning leakage: repeatedly adjusting the model based on test performance.
The correct order to prevent nearly all of these:
Split the data.
Fit any learned transformation—scaler, imputer, encoder, feature selector—using only the training data.
Apply those already-fitted transformations to the validation and test data (transform, don't refit).
Train the model on the transformed training data.
Evaluate on the untouched held-out data.
Scikit-learn's Pipeline enforces this order structurally: calling pipeline.fit(X_train, y_train) fits every step using only the data passed in, and calling pipeline.predict(X_test) or pipeline.transform(X_test) reuses the parameters already learned from training, never refitting them on the test data.
Bad example (leakage):
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X) # fit on the WHOLE dataset, before splitting
X_train, X_test, y_train, y_test = train_test_split(
X_scaled, y, test_size=0.2, random_state=42
)Corrected example:
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train) # fit only on training data
X_test_scaled = scaler.transform(X_test) # transform, don't refitTo be clear about what is not leakage: deterministic operations that use no dataset-level learned statistic—such as converting a date string into a day-of-week integer, or applying a fixed unit conversion—are not leakage, because they don't borrow any information from the dataset as a whole.
Preprocessing and Resampling
Every preprocessing step that learns something from data—an imputer's fill values, a scaler's mean and standard deviation, an encoder's category mapping, a feature selector's chosen columns, a dimensionality-reduction transform, or a target encoder's per-category statistics—should be fit exclusively on the training set and only ever applied ("transformed") on validation and test data.
Outlier handling deserves the same caution when the outlier threshold is learned from the data (for example, "anything beyond 3 standard deviations of the training mean"), since fitting that threshold on the full dataset would again mix in test information.
Oversampling and undersampling, including synthetic techniques like SMOTE, present a subtler risk. If class rebalancing happens before the split, synthetic examples generated from what later becomes test data can end up in the training set—effectively letting the model train on echoes of the test set. Resampling should normally be applied only inside the training fold (or within each cross-validation fold), never on the complete dataset prior to splitting.
Train-Test Split vs. Cross-Validation
A single holdout split is simple, fast to compute, and easy to explain, but it produces only one estimate, which can be noisy for small datasets and depends somewhat on which rows happened to land in the test set. Cross-validation addresses this by repeating the split multiple times and averaging the results.
Table: Holdout split vs. cross-validation approaches
Method | What it does | Best suited for |
Single holdout split | One train/test division | Large datasets, quick iteration |
K-fold cross-validation | K rotating splits, averaged | Small-to-medium IID datasets |
Stratified K-fold | K-fold preserving class ratios | Imbalanced classification |
Grouped cross-validation | K-fold respecting entity groups | Repeated-entity data (patients, users) |
Time-series cross-validation | Sequential expanding/rolling folds | Ordered, temporal data |
Repeated cross-validation | K-fold repeated with different splits | Small datasets needing stability |
Nested cross-validation | Outer loop for evaluation, inner loop for tuning | Reducing model-selection bias |
Cross-validation reduces the noise of a single lucky or unlucky split, but it does not automatically justify using the final test set for tuning—the same discipline of "tune on validation/cross-validation folds, evaluate once on the untouched test set" still applies. A final holdout test set and cross-validation are not mutually exclusive: many workflows use cross-validation on the training portion for model selection and hyperparameter tuning, then evaluate the finalized pipeline once on a separate test set that was never part of any fold. Nested cross-validation goes a step further, wrapping an inner loop for hyperparameter selection inside an outer loop for performance estimation, which is valuable when even ordinary cross-validation-based model selection risks optimistic bias. Repeated cross-validation—running K-fold multiple times with different random splits—can give a more stable picture of performance variability, which matters most for smaller datasets.
Evaluating Results After the Split
The right metric depends entirely on the task. For classification, common choices include accuracy, precision, recall, F1-score, ROC AUC, and PR AUC, each emphasizing a different aspect of performance—accuracy alone can be dangerously uninformative for imbalanced classes, as shown earlier. For regression, MAE, RMSE, and R² are standard, each expressing error differently, so pick the one that matches how the errors will actually be used or costed in practice.
Class distribution and real decision costs matter: in fraud detection, missing a fraud case (a false negative) is often far more costly than a false alarm, which should influence which metric and threshold get prioritized. A single point-estimate score can also be incomplete without some sense of its uncertainty—bootstrapping the test set, or examining variability across cross-validation folds, gives a rough sense of how much the reported number might shift with a different sample.
A large gap between training and test performance is a signal worth investigating—commonly overfitting, but also possibly a distribution mismatch between the two subsets or a genuine problem with the features. Conversely, a high test score does not, by itself, prove the absence of leakage or a mismatch with the deployment distribution; it should be treated as encouraging evidence, not final proof. And the test set should not be consulted repeatedly to select the "best" model among many candidates, since doing so gradually reintroduces the same bias a validation set was meant to absorb.
Common Mistakes
Mistake | Why it's harmful | Better approach |
Evaluating on training data | Hides overfitting entirely | Always evaluate on held-out data |
Preprocessing before splitting | Leaks test statistics into training | Fit preprocessing only on training data |
Tuning hyperparameters on the test set | Test score becomes optimistic | Use a validation set or cross-validation for tuning |
Randomly shuffling time-series data | Leaks future information | Use chronological or TimeSeriesSplit |
Splitting rows when entities should be grouped | Model "recognizes" entities rather than generalizing | Use GroupShuffleSplit / GroupKFold |
Ignoring class imbalance | Test set may contain almost no minority-class examples | Use stratify=y |
Assuming 80/20 is always best | Ignores dataset size and structure | Choose ratio based on data volume and problem |
Forgetting duplicates or near-duplicates | Inflates apparent test performance | Deduplicate before splitting |
Placing augmented variants of one example in both sets | Effectively duplicates training data into the test set | Split by original example ID before augmenting |
Reusing the same test set across many development rounds | Test set "wears out," score becomes biased | Reserve validation for iteration; test set sparingly |
Reporting a score with no split details | Impossible to judge reliability | State ratio, method, and sample sizes |
Treating random_state=42 as proof of correctness | Reproducible ≠ valid | Justify the splitting strategy itself |
Trusting one lucky split as definitive | Ignores sampling variability | Use cross-validation or repeated splits |
Building a test set that doesn't represent deployment | Score won't predict real performance | Match test distribution to deployment conditions |
Mixing future information into historical features | Produces temporal leakage | Compute features using only past-available data |
Choosing the Right Strategy
The decision essentially comes down to answering three questions in order: Are observations independent of each other? Does the data have a meaningful time order relative to the prediction task? Are classes balanced enough for a plain random split to be reliable?
A simplified decision guide:
Ordinary independent tabular data, balanced classes or regression target → random split (train_test_split with shuffle=True).
Imbalanced classification → stratified split (stratify=y) or StratifiedShuffleSplit / StratifiedKFold.
Repeated records per entity (patients, users, machines, authors) → GroupShuffleSplit / GroupKFold, combined with stratification if also imbalanced.
Time-ordered observations, forecasting-style task → chronological holdout or TimeSeriesSplit.
Geographic or site-based generalization needs → treat site/region as the group for a group-aware split.
Very small datasets → favor cross-validation, repeated cross-validation, or nested cross-validation over a single fixed split.
Very large datasets → a single holdout split is usually fine and computationally efficient; consider protected slice-level test sets for important subgroups.
Streaming or continuously changing data → chronological splitting plus ongoing monitoring for drift after deployment.
Competitions with a predefined holdout → follow the competition's split exactly; don't recreate your own from the training portion in a way that mismatches it.
Production systems subject to drift → periodic re-evaluation on freshly collected data, not just the original static test set.
The single unifying principle behind every branch of this guide is to match the evaluation split to the real deployment question: what will the model actually be asked to predict, for whom, and using what information, once it's live?
Small Datasets
Small datasets create specific evaluation challenges. Too few test observations make any single metric unstable; rare classes can vanish from a subset entirely; overall metrics can swing noticeably between two otherwise-reasonable splits; and reserving a large enough test set to be trustworthy can leave too little data left over for training.
Reasonable responses include using stratification where classes are involved, replacing a single split with K-fold or repeated K-fold cross-validation, applying group-aware cross-validation when entities repeat, and using nested cross-validation when both hyperparameter tuning and final evaluation need to happen on limited data. Reporting a measure of uncertainty alongside the point estimate—rather than a single confident-sounding number—is especially important here. When possible, collecting more data is the most direct fix; when not possible, it's better to be transparent about small test counts than to present a fragile estimate as settled fact. No single technique, including leave-one-out cross-validation, should be treated as automatically best for every small-data situation—the right choice still depends on computational budget and how the data is structured.
Large Datasets
Large datasets flip several of the small-data concerns. A smaller test percentage can still produce a large, statistically precise test set—90/10 on a million-row dataset still leaves 100,000 test rows. This also keeps computation manageable, since retraining or repeated cross-validation on very large datasets can be expensive.
It's still worth checking that the test set represents rare subgroups and covers the relevant temporal or geographic range, rather than assuming a large random sample automatically covers everything proportionally. Deduplication remains important even at scale, since duplicate rows can inflate apparent performance regardless of dataset size. Many mature workflows also maintain a durable, protected holdout set that is rarely touched, alongside separate stress-test or slice-evaluation sets built specifically to check performance on known-important subgroups, rather than relying on one aggregate number to represent every use case.
Reproducibility and Randomness
Random splitting produces different subsets each time it runs unless the randomness is controlled. A random seed, passed as random_state, fixes the internal random number generator so the same call produces the same split every time, which is valuable for debugging and for letting others reproduce your exact result.
Reporting the seed used improves reproducibility, but testing only a single seed can hide how much a result depends on which particular rows landed in the test set. Evaluating a model across several different, reasonable random seeds—and looking at the spread of resulting scores—can reveal whether a result is stable or was partly a matter of luck. It is invalid, however, to search across seeds and report only whichever one produced the best-looking test score; that is a form of test-set tuning by another name. Reproducible experiments and robust conclusions are related but distinct: a result can be perfectly reproducible and still be a fragile, unlucky (or lucky) draw.
Production Considerations
An offline train-test split is only useful to the extent it approximates how the model will actually encounter data after deployment. Several forces can erode that approximation over time: distribution shift in the input features, concept drift where the relationship between features and target itself changes, label definitions that get redefined by the business, and delayed labels that aren't available until long after a prediction is made (common in credit risk or long-cycle sales).
Sound practice includes monitoring performance after deployment rather than assuming the original test score holds indefinitely, setting a schedule for re-evaluation, maintaining a protected benchmark set that isn't touched by day-to-day experimentation, backtesting changes before rollout where feasible, and planning for periodic retraining. If the original test set has influenced many rounds of decisions over a project's lifetime, it's worth treating it as compromised and collecting a fresh evaluation sample rather than continuing to lean on the same one indefinitely.
Best-Practice Checklist
Before training a model, confirm:
The prediction unit is clearly defined (a row, a patient, a session, a time window).
The deployment scenario is understood (what data will be available at prediction time).
Independence assumptions have been checked, not assumed.
Time ordering has been identified, if present.
Groups or repeated entities have been identified.
Duplicates and near-duplicates have been checked and removed.
Class balance has been examined.
The split proportions are justified by dataset size and structure.
Preprocessing is fit only on training data.
A validation strategy (holdout validation set or cross-validation) is in place for tuning.
The evaluation metric matches the task and its real costs.
The random seed is recorded.
The test set is protected from repeated use.
The whole pipeline is reproducible.
The split method, metric, and limitations are reported alongside the score.
A plan exists for monitoring performance after deployment.
FAQ
What is a train-test split in simple terms?
It's dividing your dataset into two parts: one the model learns from (training set), and one it's tested on afterward (test set) to see how well it handles data it has never seen.
Why is train-test splitting necessary?
Without it, you can only measure how well a model fits data it already memorized, which tells you almost nothing about whether it will work on new, real-world cases. Splitting creates a fair, unseen test.
What is the best train-test split ratio?
There isn't one universal best ratio. Common choices are 70/30, 80/20, or 90/10, and the right one depends on your dataset size, class balance, and whether you're also using cross-validation.
Is 80/20 always the best split?
No. 80/20 is a popular default, but it can leave too little data for training on small datasets, or waste precision on very large ones. Base the ratio on your specific data, not habit.
What does random_state=42 mean?
It fixes the random number generator so the split is identical every time the code runs. The number 42 has no special statistical meaning—any fixed integer works the same way for reproducibility.
Should you shuffle data before splitting?
Usually yes, for independent tabular data, since scikit-learn's shuffle=True default helps avoid accidental ordering bias. For time-series data, shuffling is usually wrong, because it lets future information leak into training.
When should you use stratify=y?
Use it for classification problems, especially with imbalanced classes, so the training and test sets keep roughly the same class proportions as the full dataset.
Can you stratify regression data?
Not in the standard sense scikit-learn's stratify parameter is built for, since it expects discrete class labels. Continuous regression targets aren't stratified the same way classification labels are.
What is the difference between validation and test data?
Validation data is used repeatedly during development to tune hyperparameters and compare models. Test data is reserved for one final, unbiased performance estimate after development is finished.
Can the test set be used more than once?
Ideally it's used once. Repeated use for tuning or model selection gradually biases the score, since decisions start being optimized against that specific test sample.
Should preprocessing happen before or after splitting?
After splitting. Fit any learned preprocessing step—scalers, imputers, encoders—only on the training data, then apply that same fitted transformation to the test data.
How do you split time-series data?
Typically by chronological order, training on earlier data and testing on later data, using tools like scikit-learn's TimeSeriesSplit rather than a random shuffle.
How do you split records belonging to the same person or group?
Use a group-aware splitter, such as GroupShuffleSplit or GroupKFold, which keeps every record from one entity entirely in either the training set or the test set, never both.
Is cross-validation better than a train-test split?
They serve different purposes. Cross-validation gives a more stable performance estimate, especially for smaller datasets, but a protected final test set is still valuable for an honest, one-time check.
What should you do with a very small dataset?
Favor cross-validation, repeated cross-validation, or nested cross-validation over a single fixed split, and be cautious about drawing firm conclusions from a handful of test examples.
How can you detect data leakage?
Watch for suspiciously high scores, check whether any preprocessing was fit before splitting, confirm entities and duplicates aren't crossing between train and test, and verify that time-ordered data wasn't shuffled.
Key Takeaways
A train-test split exists to estimate how a model will perform on data it hasn't seen, not to measure how well it memorized training data.
The test set must stay untouched until a final evaluation, or its score stops being trustworthy.
The right splitting method—random, stratified, grouped, or chronological—should match the actual structure of your data.
Leakage, in any of its many forms, is one of the most common reasons reported performance doesn't hold up in production.
Validation sets and cross-validation exist to guide development decisions without contaminating the final test evaluation.
Fixing a random seed makes a split reproducible, but reproducibility alone doesn't make a split statistically sound.
A good test set should represent the conditions the model will actually face after deployment.
No single split ratio, splitting method, or cross-validation scheme is universally correct for every dataset.
Actionable Next Steps
Identify the real prediction unit for your problem—row, patient, session, or time window.
Check the dataset for groups, timestamps, duplicates, and class imbalance before choosing a split method.
Choose a splitting strategy that matches what you found—random, stratified, grouped, or chronological.
Reserve a protected test set and avoid touching it during development.
Build preprocessing and modeling steps into a single Pipeline to prevent leakage.
Use a validation set or cross-validation for all tuning and model-comparison decisions.
Evaluate on the test set only once the modeling workflow is settled.
Record the split method, evaluation metric, and random seed used.
Check performance across important subgroups, not just the overall average.
Plan how you'll monitor performance and refresh the evaluation data after deployment.
Glossary
Cross-validation: A technique that splits data into multiple folds, training and evaluating repeatedly across different subsets to produce a more stable performance estimate.
Data leakage: Information from outside the legitimate training scope influencing a model or its evaluation, producing inflated results.
Distribution shift: A change in the statistical properties of input data over time, which can reduce a deployed model's accuracy.
Feature: An individual measurable input variable used by a model to make predictions.
Generalization: A model's ability to perform accurately on new, unseen data beyond its training examples.
Grouped split: A split that keeps all records belonging to the same entity in one subset, preventing leakage across related rows.
Holdout set: Data withheld from training and used for evaluation, sometimes used interchangeably with test set or validation set depending on context.
Hyperparameter: A configuration value chosen before training, rather than learned from data, such as tree depth or learning rate.
IID (independent and identically distributed): An assumption that observations don't influence each other and come from the same underlying distribution.
Label: The correct answer attached to a training example in supervised learning, also called the target.
Model selection: The process of choosing among candidate algorithms or configurations based on validation performance.
Out-of-sample data: Data that falls outside the sample used to fit a model.
Overfitting: When a model fits training data too closely, including its noise, and performs worse on new data as a result.
Pipeline: A scikit-learn object that chains preprocessing steps and an estimator so fitting only happens on training data.
Random seed: A fixed value that controls a random number generator, making a random process reproducible.
Stratification: A splitting method that preserves the proportion of each class across subsets.
Target: The variable a supervised model is trained to predict.
Temporal leakage: Future information reaching a model during training when the task requires predicting the future from the past.
Test set: A labeled subset withheld from training and used, ideally once, to produce a final, unbiased performance estimate.
Training set: The labeled data a model directly learns from.
Underfitting: When a model is too simple to capture the real patterns in data, performing poorly on both training and test data.
Validation set: A subset used during development to tune hyperparameters and compare models, separate from the final test set.
Sources & References
Scikit-learn developers. "train_test_split — scikit-learn 1.9 documentation." Scikit-learn. https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html
Scikit-learn developers. "StratifiedShuffleSplit — scikit-learn 1.9 documentation." Scikit-learn. https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedShuffleSplit.html
Scikit-learn developers. "Cross-validation: evaluating estimator performance." Scikit-learn User Guide. https://scikit-learn.org/stable/modules/cross_validation.html
Scikit-learn developers. "3.1. Cross-validation iterators with stratification based on class labels." Scikit-learn User Guide. https://scikit-learn.org/stable/modules/cross_validation.html#stratification
Scikit-learn developers. "sklearn.model_selection.GroupShuffleSplit." Scikit-learn documentation. https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupShuffleSplit.html
Scikit-learn developers. "sklearn.model_selection.GroupKFold." Scikit-learn documentation. https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupKFold.html
Scikit-learn developers. "sklearn.model_selection.TimeSeriesSplit." Scikit-learn documentation. https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html
Scikit-learn developers. "sklearn.pipeline.Pipeline." Scikit-learn documentation. https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html
Scikit-learn developers. "Common pitfalls and recommended practices." Scikit-learn User Guide. https://scikit-learn.org/stable/common_pitfalls.html
Google for Developers. "Machine Learning Crash Course: Training, validation, and test sets." Google. https://developers.google.com/machine-learning/crash-course
Schema.org. "BlogPosting." Schema.org. https://schema.org/BlogPosting
Schema.org. "FAQPage." Schema.org. https://schema.org/FAQPage
Google Search Central. "Article structured data." Google for Developers. https://developers.google.com/search/docs/appearance/structured-data/article
