What Is a Validation Set?
- 4 hours ago
- 23 min read
Every machine learning model you've ever heard of—GPT, ResNet, XGBoost—was shaped by a process that most tutorials gloss over. It isn't just training. It's the careful, deliberate act of checking, adjusting, and re-checking a model on data it has never trained on. Skip that step, and you build a model that looks brilliant in development and falls apart in the real world. The validation set is the tool that prevents that failure. Understanding it is not optional. It is one of the most important skills in machine learning.
TL;DR
A validation set is a portion of your dataset held back from training, used during model development to tune hyperparameters and detect overfitting.
It is not the same as a test set. The test set is used once, at the very end, to estimate final performance.
Using validation data incorrectly—especially by leaking preprocessing information from it—invalidates your results.
The right split strategy (random, stratified, time-based, group-based) depends on your data structure.
For small datasets, k-fold cross-validation often replaces a single validation set.
A validation set is only useful if it reflects the distribution of data your model will face after deployment.
A validation set is a portion of a dataset set aside during model development—separate from training data—used to evaluate model performance, tune hyperparameters, and compare algorithms before final testing. It acts as a feedback signal during development, helping developers detect overfitting and make better modeling decisions without touching the test set.
Table of Contents
1. What Is a Validation Set?
A validation set is a labeled subset of your data that is kept separate from the training set and used during model development—not after it—to evaluate model performance, guide hyperparameter choices, and compare candidate models.
Here is the precise definition: a validation set is a portion of data that the model never trains on, but that the developer consults repeatedly while building and refining the model. It provides honest feedback on how well the model is learning to generalize—before you commit to a final evaluation.
It does not teach the model anything. It does not represent the final report card. It is the mirror you look into while you are still getting dressed, so you don't arrive somewhere looking wrong.
The key word is during. A test set is used after. That distinction matters enormously, and it is the source of most confusion about validation in machine learning.
2. Training Set vs. Validation Set vs. Test Set
A complete dataset in supervised machine learning is typically divided into three parts. Each part has a specific, non-interchangeable role.
Dataset Part | Main Purpose | Seen During Training? | Used for Model Selection? | When Used |
Training set | Teach the model | Yes | No | Every training iteration |
Validation set | Tune and compare models | No | Yes | Repeatedly, during development |
Test set | Estimate final real-world performance | No | No | Once, at the end |
Training set: The model learns from this. Weights, parameters, and decision boundaries are all adjusted based on training data. The model sees it many times.
Validation set: The model never updates its parameters based on this data. But the developer uses the results to adjust hyperparameters, select features, choose between models, and decide when to stop training.
Test set: This is sealed until the very end. Its sole purpose is to give an unbiased estimate of how the final model performs on data it has never seen in any form. Once you use test results to make modeling decisions, the test set stops being a test set—it becomes a validation set.
3. A Simple Analogy: Studying for an Exam
This analogy appears in Deep Learning by Goodfellow, Bengio, and Courville (MIT Press, 2016) and in Andrew Ng's Stanford CS229 course materials. It holds up because it maps perfectly to how the three sets actually function.
Training set = your practice problems. You work through them. You learn. You make mistakes and correct them.
Validation set = mock exams you take before the real exam. After studying a chapter, you test yourself with a practice test you haven't seen. If you score badly, you know you need more work on that area. You adjust your strategy. You study differently. You take another mock exam.
Test set = the real final exam. It is taken once. The result is final. Its credibility depends entirely on the fact that you never used it to guide your studying.
Now here is the critical insight: if you used the final exam questions to study—if you looked at the answers, adjusted your preparation based on them, and then took the same exam—your score would be meaningless. It would not tell you what you actually know. It would only tell you how well you memorized those specific questions.
This is exactly what happens when you use the test set for model selection. The test set no longer measures generalization. It measures memorization of the test set.
4. Why Do We Need a Validation Set?
Models can and do memorize training data. A deep neural network with millions of parameters can achieve near-perfect accuracy on training data by fitting to noise—specific, accidental patterns that exist in the training set but nowhere else. Without a separate evaluation, you have no way to detect this.
Training performance is misleading. It is not a reliable indicator of how the model will perform on new data. A training accuracy of 99% means the model learned the training data well. It says nothing about generalization.
Validation detects overfitting early. If training accuracy climbs but validation accuracy stalls or drops, you know the model is memorizing rather than generalizing.
Validation enables hyperparameter tuning. Learning rate, tree depth, regularization strength, dropout rate, number of layers—none of these are learned by the model. They are chosen by you. The validation set is the only fair way to compare different hyperparameter configurations.
Validation supports model comparison. If you are evaluating logistic regression, a random forest, and a gradient boosted tree, you compare them on the validation set. You cannot compare on training data, because more complex models always win there—whether or not they actually generalize better.
Validation determines when to stop training. In neural networks, you can often see the exact epoch at which the model stops improving on validation data. This is the basis for early stopping.
Validation protects the test set. Every time you look at test set results and make a modeling decision based on them, you lose a little of the test set's reliability. The validation set absorbs all that iteration. The test set stays clean.
5. Where a Validation Set Fits in the Machine Learning Workflow
The validation set is not a step you do once. It is part of a development loop that repeats until you are satisfied with your model.
A standard workflow looks like this:
Collect and label data.
Clean and prepare data (handle missing values, encode categories, normalize features).
Split data into training, validation, and test sets—in that order, before any other processing.
Train the model on the training set.
Evaluate on the validation set.
Analyze results: is accuracy acceptable? Is overfitting present? Are certain classes being misclassified?
Adjust: change hyperparameters, modify architecture, add regularization, engineer new features.
Repeat steps 4–7 until performance is satisfactory.
Evaluate the final, chosen model once on the test set.
Report results, deploy, or both.
The validation set is active in steps 5, 6, 7, and 8. The test set appears only in step 9. That is not a coincidence. That is the design.
6. Example: Splitting a Dataset
Suppose you have a dataset of 10,000 labeled customer records for a churn prediction model.
A common 70/15/15 split would allocate:
Split | Percentage | Row Count | Purpose |
Training | 70% | 7,000 | Model training |
Validation | 15% | 1,500 | Hyperparameter tuning, model selection |
Test | 15% | 1,500 | Final unbiased evaluation |
During development, you train on the 7,000 rows. After each configuration change, you evaluate on the 1,500-row validation set. You might do this 20, 50, or 200 times.
When you have finalized your model—fixed your architecture, your hyperparameters, your features—you run one evaluation on the 1,500-row test set. That number is what you report.
The test set never appears in the loop. The validation set lives in it.
7. How Validation Sets Help With Hyperparameter Tuning
There are two categories of values in a machine learning model:
Parameters are learned automatically during training. In a neural network, weights are parameters. In a linear model, coefficients are parameters. The training algorithm adjusts them.
Hyperparameters are set by the developer before training begins. The learning rate, the maximum depth of a decision tree, the number of layers in a neural network, the regularization coefficient, the batch size—these are hyperparameters.
No algorithm learns hyperparameters for you. You must choose them. The question is: choose them based on what?
The answer is the validation set.
Consider a decision tree classifier with four candidate values for maximum depth: 3, 5, 10, and 20.
Max Depth | Training Accuracy | Validation Accuracy |
3 | 74% | 73% |
5 | 82% | 81% |
10 | 91% | 80% |
20 | 99% | 76% |
If you selected based on training accuracy alone, you would choose depth 20. But that model has clearly overfit—it performs 23 percentage points worse on validation data than on training data.
The validation set reveals this. It tells you that depth 5 generalizes best, even though it doesn't "win" on training data.
This is the entire point of hyperparameter tuning with a validation set: you are not optimizing for training performance. You are finding the configuration that performs best on data the model has never trained on.
8. How a Validation Set Helps Detect Overfitting
Overfitting happens when a model learns the training data so well—including its noise and accidental patterns—that it performs poorly on any new data. The model has memorized the training set rather than learning the underlying pattern.
The clearest signal of overfitting is a large gap between training performance and validation performance.
Metric | Training | Validation | Interpretation |
Accuracy | 99% | 72% | Severe overfitting |
Accuracy | 87% | 85% | Healthy, minimal gap |
Loss | 0.04 | 0.81 | Severe overfitting |
Loss | 0.22 | 0.25 | Healthy, minimal gap |
In deep learning, overfitting often appears as a divergence between training loss and validation loss over epochs. Training loss continues to fall. Validation loss falls for a while, then stops falling—or starts rising.
This is the visual signature of overfitting: a V-shape in validation loss, with the minimum marking the optimal stopping point.
When you see this, your responses include: adding dropout, increasing regularization (L1/L2), reducing model complexity, adding more training data, using data augmentation, or applying early stopping.
None of these interventions are possible without a validation set to diagnose the problem in the first place.
9. Can a Validation Set Help Detect Underfitting?
Yes. Underfitting is the opposite failure mode: the model is too simple, or hasn't trained enough, to capture the pattern in the data. It performs poorly on both training and validation data.
Metric | Training | Validation | Interpretation |
Accuracy | 62% | 60% | Underfitting |
Accuracy | 58% | 57% | Underfitting |
Loss | 0.85 | 0.87 | Underfitting |
Common causes of underfitting:
Model is too simple for the problem (e.g., linear model for highly nonlinear data).
Too few features, or the wrong features.
Too much regularization—the model is penalized too heavily for complexity.
Not enough training time or iterations.
Poor data quality or insufficient labeled data.
The validation set confirms underfitting by showing that both training and validation scores are low. If only validation were low, you'd suspect overfitting. When both are low, the model itself needs work.
10. Validation Loss and Validation Accuracy
These are two different ways of measuring how well your model performs on the validation set.
Validation accuracy is straightforward: what percentage of validation predictions are correct? It is easy to interpret, especially for classification problems with balanced classes.
Validation loss is more nuanced. It measures the raw difference between predicted probabilities and true labels, using a mathematical function like cross-entropy or mean squared error. Loss can change even when accuracy stays the same.
Here is why loss matters: suppose your model classifies 80 examples correctly in both epoch 10 and epoch 20. Accuracy has not changed. But in epoch 20, the model's predicted probabilities for correct classes are slightly more extreme—it is more confident in its right answers. Loss decreases.
Loss is a more sensitive signal. It often detects improvement or degradation before accuracy does. This is why loss curves are preferred for monitoring training in deep learning frameworks like TensorFlow and PyTorch.
For best practice, track both. Use accuracy when you need an interpretable business metric. Use loss when you are diagnosing training dynamics.
11. Validation Set vs. Test Set: The Most Important Difference
This distinction is the single most misunderstood concept in machine learning model evaluation.
The validation set and test set both contain data the model has not trained on. That similarity makes them look interchangeable. They are not.
The validation set is used many times. Every hyperparameter change, every feature addition, every architecture modification is evaluated on the validation set. In a serious project, you might run hundreds of evaluations against it.
The test set is used once. It is sealed until the moment you want to report or deploy the final model. Then it is opened, evaluated, and closed.
The validation set influences modeling decisions. By definition, you use validation results to change the model. That means the model—through the developer—has been implicitly tuned to the validation set.
The test set provides a true blind evaluation. Because no modeling decisions are made based on it, the test set result is an honest estimate of real-world performance.
Once you look at test results and adjust the model accordingly—even once—you have contaminated the test set. It is now functioning as a second validation set. Your final performance estimate is no longer reliable.
This matters practically. Researchers who use test results to guide improvements tend to report performance numbers that are systematically too optimistic. This is one of the reasons ML papers sometimes fail to replicate in production.
12. Common Train-Validation-Test Split Ratios
There is no universal correct split ratio. The right ratio depends on dataset size, class balance, and how much data the model needs to train effectively.
Split Ratio | Training | Validation | Test | Best For |
60/20/20 | 60% | 20% | 20% | Small-to-medium datasets |
70/15/15 | 70% | 15% | 15% | Medium datasets, general use |
80/10/10 | 80% | 10% | 10% | Moderate to large datasets |
90/5/5 | 90% | 5% | 5% | Very large datasets (millions of rows) |
For small datasets (fewer than a few thousand rows), even a 20% validation set may be too small to give a reliable estimate. Cross-validation is usually the better option here.
For very large datasets (millions of rows), a 5% validation split is often more than adequate. A 5% cut of 5 million rows is 250,000 examples—large enough for stable estimates.
For imbalanced datasets, the split ratios matter less than ensuring enough minority-class examples are present in the validation and test sets. Stratified splitting (discussed below) handles this.
13. How to Create a Validation Set
There are four main strategies. The right one depends on the structure of your data.
Random Split
Randomly assign each row to training, validation, or test. Works well when:
Data is independent and identically distributed (i.i.d.).
There is no meaningful time order.
Rows are not grouped by user, household, device, or any other entity.
Random splits can fail when data has structure that random assignment breaks—most commonly, time-ordering and grouped observations.
Stratified Split
Preserve the class distribution across all three splits. When your dataset has 5% fraud cases, you want approximately 5% fraud cases in the training, validation, and test sets—not 1% in one and 9% in another by chance.
Stratified splitting is the standard practice for imbalanced classification. Scikit-learn supports it natively via the stratify parameter in train_test_split.
Time-Based Split
For any data with a meaningful temporal order—financial time series, log data, forecasting problems, sales history—validation data must come after training data chronologically.
Example:
Training data: January–August
Validation data: September–October
Test data: November–December
Random splitting a time series leaks future information into training. The model learns from future data and then is evaluated on "past" data, producing falsely optimistic results.
Group-Based Split
When multiple rows belong to the same entity—user, patient, document, household, device—keep entire groups together in one split.
If the same user appears in both training and validation, the model may appear to generalize by simply recognizing patterns specific to that user. The validation result is inflated.
Scikit-learn's GroupShuffleSplit and GroupKFold handle this case.
14. Validation Sets and Data Leakage
Data leakage is when information that should not be available at prediction time—or information from the validation/test set—is accidentally incorporated into model training or evaluation.
Leakage is the single most dangerous mistake in machine learning workflows. It produces validation results that look excellent but disintegrate in deployment.
Common sources of leakage in validation:
Preprocessing before splitting. If you standardize (subtract mean, divide by standard deviation) the entire dataset before splitting, the mean and standard deviation you computed include information from the validation and test sets. The model has, in a subtle way, already "seen" those sets.
Feature selection before splitting. If you select features based on correlation with the target using the full dataset, you are using validation and test information to choose features.
Duplicate rows across splits. If the same row appears in both training and validation, the model has trained on that exact example it is being evaluated on.
Target leakage. A feature that is computed after the target event occurs—for example, including "account closed = yes" as a feature when predicting churn—gives the model information it wouldn't have at prediction time.
Group leakage. The same user in training and validation (described above).
The fix is always the same: split first, preprocess after, using only information from the training set.
15. How to Preprocess Data Correctly With a Validation Set
The correct order is: split → fit → transform.
Step 1: Split the data into training, validation, and test sets. No preprocessing yet.
Step 2: Fit preprocessing steps on the training set only. Compute the mean and standard deviation for standardization using only training rows. Fit a vocabulary on training text only. Learn imputation values from training data only.
Step 3: Transform training, validation, and test sets using the values learned in step 2.
This process ensures that no information from validation or test data leaks into the model.
Example for standardization:
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train) # Fit on training only
X_val_scaled = scaler.transform(X_val) # Apply same transform
X_test_scaled = scaler.transform(X_test) # Apply same transformThe fit_transform call on training data learns the mean and standard deviation. The transform calls on validation and test apply those same values—they learn nothing new.
Using scikit-learn Pipeline objects is the cleanest way to enforce this pattern, particularly when working across different team members or re-running notebooks.
16. Validation Set vs. Cross-Validation
A single validation split has a weakness: the results depend on which examples ended up in each split. If you got unlucky and the validation set contains harder-than-average examples, your validation score will be lower than it should be.
K-fold cross-validation addresses this by using multiple validation splits.
The dataset is divided into k equal folds. The model is trained k times. Each time, one fold is held out as validation and the remaining k−1 folds are used for training. The validation scores are averaged across all k runs.
Method | How It Works | Advantages | Disadvantages | Best For |
Single validation set | One fixed holdout split | Simple, fast | High variance for small datasets | Large datasets |
K-fold cross-validation | K rounds of held-out folds | Lower variance, uses all data | More compute, slower | Small to medium datasets |
Stratified K-fold | K-fold preserving class balance | Low variance + class balance | Slower | Imbalanced classification |
Time-series split | Expanding window in order | Respects temporal structure | Fewer evaluation folds | Time series |
A critical point: even when using cross-validation, you still need a separate test set. Cross-validation is a validation strategy. The test set is for final reporting.
As stated in the scikit-learn documentation (scikit-learn.org, accessed 2025): "Cross-validation gives you an unbiased estimate of model performance during model selection, but a held-out test set is still recommended for the final evaluation" (scikit-learn.org/stable/modules/cross_validation.html).
17. Validation Sets in Deep Learning
In deep learning—neural networks trained with gradient descent over many epochs—the validation set plays a continuous role throughout training, not just at the end.
After each epoch (one full pass through the training data), the model evaluates on the validation set. The developer watches both curves:
Training loss: should decrease steadily.
Validation loss: should also decrease, but only up to a point.
Early stopping is the technique of halting training when validation loss stops improving for a defined number of consecutive epochs (called the patience). This prevents overfitting automatically.
TensorFlow/Keras supports this natively:
from tensorflow.keras.callbacks import EarlyStopping
early_stop = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)
model.fit(X_train, y_train, validation_data=(X_val, y_val), callbacks=[early_stop])The patience=5 argument means training stops if validation loss does not improve for 5 consecutive epochs. restore_best_weights=True reverts the model to its best-performing checkpoint.
This is only possible because the validation set is evaluated separately from training. The model is not gradient-descending toward the validation set—it is being checked against it.
The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) dataset, one of the most influential benchmarks in deep learning history, used a publicly defined 50,000-image validation split (separate from the 1.2 million training images) specifically to allow fair hyperparameter comparison between submissions (Russakovsky et al., IJCV, 2015).
18. Validation Sets for Time Series
Time series data breaks the assumption that data points are independent. What happened yesterday informs what happens today. This means standard random splits are almost always wrong.
The correct principle for time series validation: validation data must come chronologically after all training data.
Window | Training Period | Validation Period |
Walk-forward window 1 | Jan–Jun | Jul–Aug |
Walk-forward window 2 | Jan–Aug | Sep–Oct |
Walk-forward window 3 | Jan–Oct | Nov–Dec |
This expanding-window approach (also called walk-forward validation) simulates how the model would be deployed in production—always trained on the past, always evaluated on a future period it has not seen.
Rolling-window validation uses a fixed training window size instead of expanding, which is appropriate when older data may be less relevant (for example, in rapidly changing markets).
Random splitting a time series dataset allows training data to include examples from after the validation period. This is pure future leakage. The model appears to predict the future but is really interpolating between future values it already "knows."
19. Validation Sets for Imbalanced Datasets
In fraud detection, medical diagnosis, equipment failure prediction, and many other domains, the positive class may represent 1%–5% of the data. Randomly splitting a dataset with 2% positive examples could produce a validation set with zero positive examples—making it completely useless for evaluation.
Stratified splitting ensures that each split contains approximately the same proportion of each class as the full dataset.
Beyond the split strategy, evaluation metrics also matter. For imbalanced validation:
Accuracy is often misleading. A model that always predicts "not fraud" achieves 99% accuracy on a 1%-fraud dataset—while detecting nothing.
Precision, recall, and F1-score reflect performance on the minority class specifically.
ROC-AUC and PR-AUC (precision-recall AUC) measure performance across different decision thresholds.
Confusion matrix shows exactly where the model makes mistakes.
Your validation set must contain enough minority-class examples to make these metrics meaningful. For very rare events, you may need to ensure the validation set contains at minimum 50–100 positive examples, even if this requires a larger validation proportion than you would otherwise use.
20. Common Mistakes to Avoid
Using the test set as a validation set. Every time you look at test results and use them to guide modeling decisions, you corrupt the test set. This is the most consequential mistake in ML evaluation.
Making the validation set too small. A validation set with 100 examples is statistically unstable. A 2% change in accuracy could be within the margin of chance. Use cross-validation for small datasets.
Random splitting time series data. This causes future leakage and produces results that cannot replicate in deployment.
Preprocessing before splitting. Fitting a scaler, encoder, or imputer on the full dataset before splitting leaks validation and test statistics into training. Always split first.
Ignoring class imbalance in splits. Without stratification, small classes may be very unevenly distributed across splits, producing unstable and misleading validation results.
Splitting records with shared IDs across datasets. If user A appears in both training and validation, performance is inflated because the model has already seen that user's patterns.
Treating validation performance as guaranteed production performance. Validation performance is an estimate. If production data has a different distribution (distribution shift), validation results may not hold.
Choosing metrics that don't match the real goal. If the business goal is minimizing false negatives (e.g., missing a disease case), optimizing validation accuracy is the wrong metric. Use recall or F1.
21. Best Practices
Always split before preprocessing. This is not optional.
Use stratified splits for classification with class imbalance.
Use chronological splits for time-dependent data.
Use group splits when records share an ID or entity.
Track both training and validation metrics simultaneously to catch overfitting early.
Never use the test set more than once. If you re-evaluate it after changes, it is no longer a test set.
Document your split strategy. The split logic should be reproducible. Set random_state in scikit-learn to a fixed integer.
Use cross-validation for datasets under ~5,000 rows.
Consider validation performance in the context of deployment. Does your validation data resemble what the model will encounter in production?
After finalizing all modeling decisions, you may optionally retrain on training + validation combined before final evaluation on the test set—but only if model choices are completely locked.
22. Simple Python Example: Creating a Validation Set
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_classification
# Generate a synthetic classification dataset
X, y = make_classification(
n_samples=10000,
n_features=20,
n_informative=10,
n_classes=2,
weights=[0.85, 0.15], # Imbalanced: 85% class 0, 15% class 1
random_state=42
)
# Step 1: Split into training (70%) and temp (30%)
X_train, X_temp, y_train, y_temp = train_test_split(
X, y,
test_size=0.30,
random_state=42,
stratify=y # Preserve class distribution
)
# Step 2: Split temp into validation (15%) and test (15%)
X_val, X_test, y_val, y_test = train_test_split(
X_temp, y_temp,
test_size=0.50,
random_state=42,
stratify=y_temp # Preserve class distribution in both halves
)
print(f"Training samples: {len(X_train)}") # ~7,000
print(f"Validation samples: {len(X_val)}") # ~1,500
print(f"Test samples: {len(X_test)}") # ~1,500Line-by-line explanation:
make_classification generates a synthetic labeled dataset with intentional class imbalance (15% positive class).
The first train_test_split divides the data into 70% training and 30% temporary. stratify=y ensures both halves maintain the 85/15 class ratio.
The second train_test_split divides the 30% temporary set equally into validation and test, each 15% of the original. Again, stratify=y_temp preserves class balance.
random_state=42 makes the split reproducible. Use any integer—the key is consistency.
After this, you would fit your scaler or encoder on X_train only, then apply the same transform to X_val and X_test.
23. Frequently Asked Questions
What is a validation set in simple terms?
A validation set is a portion of your data that is not used for training but is evaluated during model development to check performance and guide decisions. It gives you honest feedback while you are still building the model.
Is a validation set the same as a test set?
No. The validation set is used many times during development. The test set is used once, at the very end, to report final performance. Using the test set for model selection corrupts its ability to provide an unbiased estimate.
Why not just use the training set to evaluate the model?
Training performance measures how well the model memorized training data, not how well it generalizes to new data. A model can achieve 100% training accuracy while being completely useless on new examples.
Why not just use the test set for tuning?
Every time you look at test results and adjust the model, you are implicitly fitting the model to the test set. The test set loses its objectivity. This is how ML research results get inflated and fail to replicate.
How much data should be in the validation set?
Enough to give statistically stable estimates—typically 10%–20% of the full dataset. For very large datasets (millions of rows), 5% or less is fine. For small datasets (a few thousand rows), use cross-validation instead.
Do I always need a validation set?
In practice, yes—if you are doing any hyperparameter tuning or model selection, you need a validation set (or cross-validation). The only exceptions are exploratory analyses where no model decisions need to be made.
What is the difference between validation and cross-validation?
A single validation set is one fixed holdout. Cross-validation repeats the holdout process k times with different subsets each time and averages the results. Cross-validation gives a more stable estimate but requires more computation.
Can validation data be used for training later?
Only after all modeling decisions are finalized. Some practitioners retrain on training + validation data before deploying or final testing, since more data typically improves performance. But this must happen after all hyperparameters and architectures are locked.
What does high training accuracy but low validation accuracy mean?
Overfitting. The model has memorized training data but fails to generalize. Solutions include reducing model complexity, adding regularization, using dropout (for neural nets), adding more training data, or stopping training earlier.
What does low training accuracy and low validation accuracy mean?
Underfitting. The model is not complex enough or has not been trained adequately. Solutions include using a more expressive model, adding useful features, reducing regularization, or training for more iterations.
What is validation loss?
Validation loss is the value of the loss function (cross-entropy, MSE, etc.) computed on the validation set. It measures how wrong the model's predictions are. It is more sensitive than accuracy and is the preferred signal for monitoring training dynamics.
What is validation accuracy?
Validation accuracy is the percentage of correct predictions on the validation set. It is easy to interpret but can be misleading for imbalanced classes.
How do validation sets prevent overfitting?
They don't prevent it directly, but they reveal it. When validation performance diverges from training performance, you know overfitting is occurring. You can then apply countermeasures: regularization, early stopping, reducing model complexity.
What is the best split between training, validation, and test data?
Depends on dataset size. For medium datasets, 70/15/15 or 60/20/20 are common starting points. For very large datasets, 80/10/10 or 90/5/5 are reasonable. For small datasets, use cross-validation.
How should validation be done for time series?
Always use chronological splits. Validation data must come from a later period than training data. Random splits leak future information and produce falsely optimistic results.
What is data leakage in validation?
Data leakage happens when information from the validation or test set—or from the future, or from related records—is accidentally included in model training or evaluation. Leakage inflates validation performance and leads to poor real-world results.
Should preprocessing happen before or after splitting the data?
Always after splitting. Fit all preprocessing transformers (scalers, encoders, imputers) on training data only. Apply the learned transformation to validation and test data. Preprocessing before splitting leaks information.
Key Takeaways
A validation set is a labeled subset held out from training and used repeatedly during model development to tune, compare, and refine models.
The three-way split—training, validation, test—gives each part a distinct and non-interchangeable role. Mixing roles corrupts results.
The validation set protects the test set by absorbing all development iteration. The test set provides one final, unbiased estimate.
Data leakage—especially from preprocessing before splitting—is the most silent and damaging validation error. Always split first, then preprocess.
The right split strategy (random, stratified, time-based, group-based) depends entirely on the structure of your data, not just the size.
Validation accuracy and validation loss are complementary signals. Loss detects problems that accuracy misses.
Cross-validation is the correct alternative to a single validation set when data is too small for stable holdout estimates.
A good validation set is representative, leak-free, appropriately sized, and structurally matched to the real deployment distribution.
Validation performance is an estimate, not a guarantee. Distribution shift between development and production can invalidate even well-designed validation results.
Building reliable ML models is not just about training—it is about evaluation design. The validation set is the foundation of that design.
Actionable Next Steps
Audit your current workflow. Are you preprocessing before splitting? Fix it now. Refit all transformers on training data only.
Check your split strategy. Do you have time-ordered data? Use chronological splits. Do you have grouped records (same user appearing multiple times)? Use group splits.
Add stratification to classification splits. If your target is imbalanced, add stratify=y to train_test_split.
Lock your test set. If you have looked at test results and adjusted your model, admit that your test set has become a second validation set. Create a new test split.
Track both training and validation metrics per epoch in neural network training. Watch for divergence—it is your early warning for overfitting.
Consider cross-validation if your dataset has fewer than 5,000 rows. A single split on a small dataset is statistically unreliable.
Match validation metrics to your actual goal. If you care about recall, evaluate recall on the validation set—not just accuracy.
Document your split strategy in code comments or a project README. Record the split ratios, the random seed, and the method used.
Glossary
Validation set: A labeled subset of data held out from training and used during model development to guide hyperparameter tuning, model selection, and overfitting detection.
Training set: The portion of data used directly to train the model—to update weights, fit parameters, and learn patterns.
Test set: A portion of data reserved for a single final evaluation after all modeling decisions are complete. Not used for training or development.
Overfitting: When a model performs well on training data but poorly on new data, because it has memorized training patterns (including noise) rather than learning generalizable ones.
Underfitting: When a model performs poorly on both training and validation data, usually because it is too simple or hasn't been trained enough.
Hyperparameter: A configuration value set before training that controls model behavior—e.g., learning rate, tree depth, dropout rate. Not learned by the model itself.
Data leakage: When information from outside the training set (from validation, test, or future data) is accidentally used to train or evaluate the model, producing inflated and misleading results.
Stratified split: A splitting method that preserves the proportion of each class across all splits, important for imbalanced classification problems.
Cross-validation (k-fold): A validation method that trains and evaluates the model k times on different data splits, averaging results for a more stable estimate.
Early stopping: A training technique that halts model training when validation performance stops improving, used to prevent overfitting in neural networks.
Distribution shift: When the statistical distribution of data in production differs from the distribution in the development dataset, causing models to perform worse than validation results suggest.
Validation loss: The value of the loss function computed on the validation set, measuring prediction error. More sensitive than accuracy for monitoring training.
Validation accuracy: The percentage of correct predictions on the validation set. Easy to interpret but potentially misleading for imbalanced classes.
References
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. https://www.deeplearningbook.org/
Scikit-learn developers. (2025). Cross-validation: evaluating estimator performance. scikit-learn.org. https://scikit-learn.org/stable/modules/cross_validation.html
Scikit-learn developers. (2025). sklearn.model_selection.train_test_split documentation. scikit-learn.org. https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html
TensorFlow team. (2025). EarlyStopping callback. TensorFlow documentation. https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping
Russakovsky, O., Deng, J., Su, H., et al. (2015). ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3), 211–252. https://link.springer.com/article/10.1007/s11263-015-0816-y
Google Developers. (2025). Training and Test Sets: Splitting Data. Google Machine Learning Crash Course. https://developers.google.com/machine-learning/crash-course/training-and-test-sets/splitting-data
Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
Ng, A. (2018). Machine Learning Yearning. deeplearning.ai. https://www.deeplearning.ai/machine-learning-yearning/
