What Is a Decision Boundary in Machine Learning? Complete Guide 202
- Jul 16
- 32 min read
Picture a scatter plot of emails, each dot colored by whether it was spam or not spam, based on two simple measurements like the number of exclamation marks and the number of links. Somewhere on that plot, there is an invisible line, curve, or shape that separates the red dots from the blue ones — and every time a new email lands on one side or the other, the model calls it spam or not spam. That invisible line is the decision boundary, and understanding how it forms, bends, and moves is the fastest way to actually understand machine learning classification rather than just running code that produces one.
TL;DR
A decision boundary is the exact place in feature space where a classifier's predicted class switches from one label to another.
In two dimensions it usually looks like a line or curve; in higher dimensions it becomes a surface, and it is not always a single connected shape.
Different algorithms produce very different boundary shapes — logistic regression draws a hyperplane, k-nearest neighbors carves jagged local regions, and decision trees stack rectangular blocks.
A decision boundary is not the same thing as a decision threshold — moving the threshold on a probability or score can shift the boundary without retraining the model at all.
A boundary that fits training data perfectly is not automatically a good boundary; overly twisted boundaries often overfit and fail on new data.
The Bayes decision boundary is the theoretical best-possible boundary given the true class distributions, and every real classifier is only an approximation of it.
What Is a Decision Boundary in Machine Learning?
A decision boundary in machine learning is the surface in feature space where a classifier's predicted class changes from one category to another. Points on one side receive one label; points on the other side receive a different label. The boundary emerges from the model's learned scoring function and decision rule, not from a manually drawn line.
Table of Contents
What Is a Decision Boundary in Machine Learning?
A decision boundary is the location in a model's feature space where its predicted class flips. Every point on one side of the boundary gets one label. Every point on the other side gets a different label. The boundary itself is the set of points where the model is, in a sense, exactly torn between the two options.
This idea sits at the center of every classification problem in machine learning. Whether the model is artificial intelligence built on logistic regression, a support vector machine, a decision tree, or a neural network, the underlying question is the same: given a data point described by its features, which category does the model assign it to? The decision boundary is simply where that answer changes.
Feature space is the mathematical space defined by the variables the model uses. If a model uses two numeric measurements, feature space is a two-dimensional plane, and the boundary is usually a line or curve. If the model uses three features, feature space is three-dimensional, and the boundary is usually a surface. With more features, the boundary becomes a higher-dimensional object that cannot be drawn directly, though it still exists mathematically and still governs every prediction the model makes [1].
Under ordinary conditions, a boundary in a feature space of dimension d is often a shape of dimension d − 1 — a line in 2D, a plane or curved surface in 3D, and so on. This is a useful rule of thumb, not an unconditional law. Real boundaries can be disconnected into several separate regions, can pinch down to isolated points, or can behave irregularly around outliers and sparse data, so it is more accurate to think of the boundary as "wherever the predicted class changes" than as a single tidy geometric object.
A Simple Visual Example
The clearest way to build intuition is a border on a map. A country's border is a specific line: cross it, and you are now governed by different laws. The analogy works well for showing that a boundary is a precise dividing line rather than a fuzzy zone. It breaks down in two ways worth flagging early. First, national borders are fixed by treaty, while a decision boundary is learned from data and can shift every time the model is retrained. Second, real decision boundaries are frequently uncertain near the line itself — unlike a legal border, which is exact even at the millimeter, a model's boundary is a best estimate that can be wrong close to the line.
Consider a two-feature loan approval example: an applicant's debt-to-income ratio on one axis and credit history length on the other. A machine learning model trained on past applications learns a rule that separates "approved" from "denied" across this two-dimensional plane. Applicants near the boundary — moderate debt, moderate history — are the hardest cases: small changes in either measurement can flip the prediction. Applicants far from the boundary — very low debt and long credit history, or the reverse — are predicted with much more consistency.
This example also shows why the boundary is not just a teaching device. Where that line sits determines who gets approved. Moving it even slightly changes real outcomes for real people, which is exactly why later sections on thresholds and evaluation matter as much as the geometry itself.
Feature Space, Decision Regions, and Class Predictions
Feature space is divided by the decision boundary into decision regions — contiguous areas where the model outputs a single class label. In binary classification, there are typically two regions, though a model can produce boundaries that carve out more than two disconnected areas even with only two classes (for example, an "approve" island surrounded entirely by "deny" territory, if the training data supports that shape).
A predicted label is simply which decision region a data point falls into. The model does not consult the boundary directly at prediction time; it computes a score or probability for the point and applies a rule. The boundary is the byproduct — the set of points where that rule is exactly on the fence.
It helps to separate four related ideas clearly:
Feature space — the coordinate system defined by the input variables.
Decision region — a connected area of feature space assigned to one class.
Predicted label — the specific class assigned to one point.
Decision boundary — the border between decision regions.
Model scores, class probabilities, and classification thresholds all feed into which decision region a point lands in, and the next section makes that relationship mathematically precise.
The Mathematical Definition of a Decision Boundary
Binary Classification
For binary classification, define a scoring function g(x), which takes a feature vector x and returns a real number. This score might be a raw model output, a distance-like quantity, or a probability. Define a threshold τ (the Greek letter tau). The classifier assigns one class when g(x) > τ, and the other class when g(x) < τ.
The decision boundary is the set of points where the score exactly equals the threshold:
B = {x : g(x) = τ}
In plain English: the boundary is every point where the model's score lands exactly on the cutoff line between the two classes. Points strictly above the threshold go to one class; points strictly below go to the other; points exactly on it are the mathematical edge case that defines the boundary itself.
Linear Boundary Equation
A common special case is the linear boundary equation:
wᵀx + b = 0
Here w is a vector of learned weights, x is the feature vector, and b is a learned bias (or intercept) term. wᵀx is the dot product — multiply each feature by its corresponding weight and add the results. When this weighted sum plus the bias equals zero, the point sits exactly on the boundary. When it is positive, the point falls on one side; when negative, the other. This equation describes a hyperplane — a flat object one dimension lower than the feature space, such as a straight line in 2D or a flat plane in 3D [2].
Logistic Regression and the 0.5 Threshold
Logistic regression models the probability of the positive class using the sigmoid function applied to a linear combination of features: P(y=1 | x) = 1 / (1 + e^(−(wᵀx + b))). A probability threshold of 0.5 corresponds exactly to wᵀx + b = 0, because the sigmoid function equals 0.5 precisely when its input is zero [3]. This makes 0.5 a mathematically convenient default, not a universally optimal one. A fraud-detection team might prefer a much lower threshold to catch more fraud at the cost of more false alarms; a spam filter might prefer a higher threshold to avoid ever blocking a real message. Changing that threshold moves the effective boundary without touching the underlying weights w and b at all.
Multiclass Classification
For more than two classes, most models compute a separate score for each class and predict whichever class has the highest score — the arg-max rule. The boundary between any two classes occurs where their scores tie and both are competitive for the top spot. This is a more subtle picture than a single equation, because with three or more classes there can be multiple such tie regions, and a point where several class scores tie simultaneously becomes a junction where several boundary segments meet. Not every multiclass system reduces neatly to one pairwise formula, and the exact structure depends on whether the model uses a shared multinomial function or a collection of pairwise or one-vs-rest classifiers [4].
Decision Boundary vs. Decision Threshold and Related Terms
These terms get used loosely, sometimes interchangeably, and that looseness causes real confusion. Here is a direct comparison.
Term | What it means | How it relates to the boundary |
Decision boundary | The geometric location in feature space where predicted class changes | The core concept — everything else refers to it |
Decision region | A connected area of feature space assigned to one class | The space on either side of the boundary |
Decision surface | Often used as a synonym for decision boundary, especially in higher dimensions | Same concept, different name |
Separating hyperplane | A flat (linear) decision boundary | A special case of a decision boundary, only when the model is linear |
Decision threshold | The cutoff value applied to a score or probability | Changing it can move the boundary without retraining |
Margin | The distance between the boundary and the nearest training points (SVM-specific) | A property of the boundary's placement, not the boundary itself |
The most important distinction is between the geometric boundary in feature space and the numerical threshold applied to a score or probability. The boundary is a shape; the threshold is a number. For many score-producing classifiers, adjusting the threshold slides the boundary through feature space without any retraining, because the underlying scoring function g(x) never changes — only the cutoff τ does.
Class probability and confidence score are also frequently conflated. A probability is a calibrated estimate that should, in principle, match observed frequencies. A confidence score or margin distance is often just a raw number from the model's internal math, and treating it as a probability without checking calibration is a common source of error (more on this in the section on margins and uncertainty).
The Bayes decision boundary, covered in depth later, is the theoretical ideal that all of these practical boundaries are trying to approximate.
Linear vs. Nonlinear Decision Boundaries
A boundary is linear when it can be described by a single linear equation like wᵀx + b = 0 — a straight line in two dimensions, a flat plane in three, and a hyperplane in higher dimensions. A hyperplane is simply the general term for a flat, one-dimension-lower object in any-dimensional space; a straight line in 2D is technically a hyperplane because it is one dimension lower than the 2D plane it lives in.
A nonlinear boundary is any boundary that cannot be written this way — a curve, a wavy line, a set of disconnected blobs, or a jagged shape. Nonlinear does not automatically mean better. A nonlinear boundary can capture genuine curved structure in the data, or it can be needlessly contorted around noise that has no real signal in it. Model flexibility — how many different shapes a model's boundary can take — is often called model capacity, and higher capacity is not a free upgrade; it comes with a greater risk of overfitting, discussed in a later section.
Feature transformations create an important twist: a boundary that is linear in one representation of the data can look completely nonlinear in another. Kernel support vector machines exploit this directly — they compute an implicit transformation of the original features into a new (often much higher-dimensional) space, fit a linear boundary there, and when that boundary is projected back down into the original feature space, it appears curved [5]. Polynomial regression features work the same way: adding x² and xy terms as inputs to a linear model lets that linear model draw curved boundaries in terms of the original x and y variables, even though the underlying equation is still linear in the expanded feature set.
How Different Machine-Learning Algorithms Form Boundaries
Logistic Regression
Logistic regression fits weights w and bias b by maximizing the likelihood of the observed labels, typically with an added regularization penalty. Its boundary is always linear in the original features (though feature engineering can bend that boundary, as noted above). It is fast, interpretable, and outputs genuinely probabilistic scores when properly regularized, but it struggles whenever classes are not close to linearly separable.
Linear Support Vector Machines
A linear SVM also learns a hyperplane, but instead of maximizing likelihood, it explicitly seeks the hyperplane that maximizes the margin — the distance between the boundary and the closest training points of each class, sometimes called support vectors [6]. This margin-maximizing objective often produces boundaries that generalize well even with limited data, though the classic hard-margin formulation assumes the classes are separable; the soft-margin version, controlled by the regularization parameter C, allows some misclassified points in exchange for a wider, more robust margin.
Kernel Support Vector Machines
Kernel SVMs replace the raw feature dot product with a kernel function — commonly the radial basis function (RBF) kernel — that implicitly measures similarity in a much higher-dimensional transformed space, without ever computing that space explicitly (the so-called "kernel trick") [5]. This lets the SVM produce smoothly curved, nonlinear boundaries in the original feature space. The RBF kernel's gamma parameter controls how tightly the boundary wraps around individual points; a high gamma can produce a boundary so contorted it overfits, while a very low gamma flattens the boundary almost back to linear.
k-Nearest Neighbors
k-nearest neighbors (k-NN) has no training phase in the usual sense; at prediction time, it looks at the k closest training points to a new point and votes on the majority class among them [7]. Its boundary is entirely local and can become highly irregular, with jagged edges that hug individual training points, especially when k is small (k=1 produces a boundary that touches every training point exactly). Increasing k smooths the boundary considerably. k-NN is simple and often surprisingly effective, but it is sensitive to feature scaling and becomes slow and less reliable as the number of features grows.
Decision Trees
A standard decision tree splits feature space one axis at a time — "is feature A greater than some value?" — producing boundaries that are axis-aligned and piecewise rectangular in the original feature space [8]. This gives trees excellent interpretability, since each split is a simple readable rule, but it also means trees can require many splits to approximate a diagonal or curved true boundary, producing a jagged, staircase-like approximation. Oblique trees, a less common variant, split on linear combinations of features rather than single features, producing non-axis-aligned boundaries, though standard implementations in most libraries use axis-aligned splits by default.
Random Forests and Gradient Boosting
Random forests average the predictions of many decision trees, each trained on a bootstrapped sample of the data and a random subset of features; gradient-boosted trees build trees sequentially, each correcting the errors of the ones before it [9]. Both ensemble methods combine many simple, axis-aligned partitions into an aggregate boundary that can approximate much smoother and more complex shapes than any single tree, at the cost of interpretability and additional hyperparameters (number of trees, tree depth, learning rate for boosting) that strongly influence the final boundary's smoothness and risk of overfitting.
LDA and QDA
Linear discriminant analysis (LDA) assumes each class follows a Gaussian distribution and that all classes share the same covariance structure; under that assumption, the boundary between any two classes reduces to a straight line or flat hyperplane [10]. Quadratic discriminant analysis (QDA) relaxes the shared-covariance assumption, allowing each class its own covariance matrix, which produces boundaries described by quadratic (curved, often ellipse-like) equations [10]. LDA tends to work well with limited data because it estimates fewer parameters; QDA can capture more realistic class shapes but needs more data to estimate each class's covariance reliably.
Naive Bayes
Naive Bayes applies Bayes' theorem with an assumption that features are conditionally independent given the class. The exact shape of its boundary depends heavily on which distribution is assumed for the features and how they are represented: Gaussian Naive Bayes can produce boundaries similar to QDA under certain conditions, while Multinomial or Bernoulli Naive Bayes (common for text data) produce boundaries that are effectively linear in the log-count or log-probability feature representation [11]. There is no single geometric shape that describes "the" Naive Bayes boundary in general.
Neural Networks
A neural network with ReLU (rectified linear unit) activations partitions the input space into a large number of linear regions, producing an overall boundary that is piecewise linear — many small flat facets stitched together into a shape that can look smooth from a distance but is technically composed of straight segments [12]. Networks with smooth activations, such as sigmoid or tanh, tend to produce genuinely smooth, curved boundaries instead. Depth and width both expand how many linear regions a ReLU network can represent, which is one reason deeper networks can fit more intricate boundaries — for better or worse, depending on regularization and data size. A single-layer perceptron, notably, can only ever produce a linear boundary and famously cannot solve the XOR problem, which requires at least one hidden layer to solve.
Multiclass logistic regression (softmax regression) extends the same linear scoring idea across many classes at once, assigning each class its own linear score function and predicting whichever class scores highest; the resulting boundaries between adjacent classes are linear segments that meet at shared junctions.
Model | Typical boundary | Main complexity control | Common strength | Common risk |
Logistic regression | Linear (hyperplane) | Regularization strength | Fast, interpretable, probabilistic | Underfits nonlinear structure |
Linear SVM | Linear, margin-maximizing | Regularization parameter C | Strong generalization with clear margin | Still linear only |
Kernel SVM (RBF) | Smooth, nonlinear | Kernel gamma, C | Captures curved structure well | Can overfit with high gamma |
k-Nearest Neighbors | Irregular, local | Number of neighbors k | Simple, no training phase | Sensitive to scaling, slow at scale |
Decision tree | Axis-aligned, piecewise rectangular | Tree depth, min samples per leaf | Highly interpretable | Prone to overfitting if deep |
Random forest / boosting | Complex aggregate of rectangles | Number of trees, depth, learning rate | Strong accuracy, robust | Less interpretable |
LDA | Linear | Shared covariance assumption | Efficient with limited data | Assumes equal covariance |
QDA | Quadratic (curved) | Per-class covariance estimate | Captures different class shapes | Needs more data per class |
Naive Bayes | Depends on distribution assumed | Choice of distribution/representation | Fast, works well with text data | Independence assumption often wrong |
Neural network (ReLU) | Piecewise linear, can appear smooth | Depth, width, regularization | Captures very complex structure | Needs more data, less interpretable |
How Training Data Shapes the Boundary
A model does not draw its boundary as a separate manual step; the boundary emerges directly from the parameters learned while minimizing a loss function on training data. The loss function measures how wrong the model's predictions are; optimization algorithms like gradient descent adjust the model's parameters — weights in logistic regression and neural networks, split points in trees — to reduce that loss.
Several factors interact here. The training samples and their labels define what "correct" means during learning. Regularization adds a penalty for overly complex parameter values, nudging the boundary toward simpler shapes even if a more contorted shape would fit the training data slightly better. Model capacity — how expressive the chosen model family is — sets an upper limit on how intricate the boundary could theoretically become. Inductive bias refers to the built-in assumptions a model makes before seeing any data (linearity, axis-alignment, locality), which shape what kinds of boundaries it can even represent. Hyperparameters, such as tree depth or the SVM's C and gamma, are chosen using a validation set rather than the training set itself, to avoid quietly overfitting the hyperparameter choice to the exact same data the final model is judged on.
How Features and Preprocessing Change the Boundary
The features fed into a model directly shape what its boundary can look like, often more than the choice of algorithm does.
Feature scaling matters enormously for distance-based methods like k-nearest neighbors and margin-based methods like SVMs, because both rely on measuring distances between points. If one feature ranges from 0 to 1 and another ranges from 0 to 100,000, the larger-scaled feature will dominate the distance calculation and distort the boundary, regardless of which feature is actually more informative.
Feature selection and engineering change which patterns the model can even represent. Removing irrelevant variables can simplify and stabilize the boundary; adding well-chosen polynomial or interaction features can let an otherwise linear model draw curved boundaries in the original variables. Redundant variables rarely help and can make some models (particularly those sensitive to collinearity) less stable.
Outliers, label noise, and measurement error can pull a boundary away from where it should genuinely sit, especially for models like SVMs and logistic regression that are influenced by every training point, or margin-sensitive models where a single mislabeled point near the boundary can shift the fitted margin substantially.
Class overlap and class imbalance both complicate the picture. Overlapping classes mean no boundary, however well-placed, can perfectly separate the data — some error is irreducible. Class imbalance can bias a boundary toward the majority class unless addressed through resampling, class weighting, or threshold adjustment.
Missing-value handling, data leakage, and distribution shift are less visible but equally serious. Leakage — accidentally including information at training time that would not be available at prediction time — can produce a boundary that looks excellent in testing but fails in deployment. Distribution shift, where the real-world data drifts away from the training distribution over time, can leave a once-accurate boundary poorly matched to current conditions. Sparse, high-dimensional data (common in text and genomics) also changes boundary behavior, since distances and margins behave differently as dimensionality grows, a topic covered further in the high-dimensional visualization section.
Crucially, the geometry visible in a two-feature plot may not represent the model's actual behavior if the model was trained on many more features — a plot showing two features while others exist tells only part of the story.
Decision Boundaries, Margins, Probability, and Uncertainty
Points near a decision boundary are, almost by definition, the model's hardest cases — small changes in the input, or small amounts of noise, can flip the prediction. This is one reason distance from the boundary is often treated as a rough proxy for confidence: farther points tend to be more consistently classified.
That said, raw distance is not automatically a calibrated probability. In a support vector machine, the margin is the geometric distance between the boundary and the nearest training points; it says something about the separation the model achieved, but it is not, by itself, a probability of correctness. Turning SVM outputs into probability-like scores typically requires an additional calibration step, such as Platt scaling [13].
Calibration is the broader concept: a model is well-calibrated if, among all the times it says "80% probability of class A," roughly 80% of those cases actually are class A. A model can have excellent accuracy while still being poorly calibrated, or vice versa. This distinction matters especially in high-stakes settings — a poorly calibrated model might report high confidence right up to the boundary, giving a false sense of certainty exactly where the model is most likely to be wrong.
"Far from the boundary" does not guarantee correctness for another reason too: distribution shift or model misspecification can leave a model confidently wrong across large stretches of feature space, particularly in regions that were poorly represented in training data. In costly or ambiguous decision regions — a borderline medical diagnosis, for example — some systems build in abstention or a human-review step, deferring the final call rather than trusting an uncertain automated boundary crossing.
How Classification Thresholds Move the Effective Boundary
Most binary classifiers output some kind of score or probability before applying a decision rule. The most common default in binary classification is a probability threshold of 0.5, which — as shown earlier — corresponds to a specific mathematical location in feature space for models like logistic regression.
Business costs frequently justify a different threshold. Consider the trade-off between false positives (incorrectly predicting the positive class) and false negatives (missing an actual positive case). In fraud detection, a missed fraud case (false negative) is often far costlier than a false alarm (false positive), which argues for a lower threshold that catches more fraud at the cost of more manual reviews. In a spam filter, a false positive — a real message wrongly binned as spam — can be more damaging to a user's trust than an occasional missed spam message, arguing for a higher threshold.
Formal tools help choose a threshold deliberately rather than defaulting to 0.5. Precision measures what fraction of positive predictions were correct; recall (also called sensitivity) measures what fraction of actual positives were caught; specificity measures what fraction of actual negatives were correctly identified. The ROC curve plots the true positive rate against the false positive rate across every possible threshold, and the precision–recall curve does the analogous plot for precision and recall — both let a practitioner pick a threshold suited to a specific cost structure rather than accepting a generic default [14]. Under severe class imbalance, precision–recall curves are usually more informative than ROC curves, because ROC curves can look deceptively good even when the minority class is handled poorly.
It is worth restating clearly: threshold tuning and probability calibration are related but different tasks. Calibration adjusts what a probability means; threshold tuning decides what to do with a probability once it is trustworthy. A well-calibrated model can still need careful threshold selection, and threshold tuning cannot fix a badly miscalibrated model on its own.
Multiclass Decision Boundaries
With more than two classes, feature space divides into multiple decision regions rather than just two. Consider a simple three-class example: classifying flowers into three species based on petal length and petal width. The resulting plot typically shows three regions meeting near a shared point or short shared boundary segment, rather than one single dividing line.
Several strategies handle multiclass problems in practice. One-vs-rest (OvR) trains one binary classifier per class, each distinguishing that class from all others combined, and predicts whichever classifier is most confident. One-vs-one (OvO) trains a separate binary classifier for every pair of classes and combines their votes. Multinomial (softmax) models instead compute all class scores jointly within a single unified model, which often produces more consistent and typically more calibrated behavior than combining several independent binary classifiers [4].
Ties matter here: multiclass boundaries occur specifically where two or more class scores are equal and jointly maximal. The arg-max rule — predict whichever class has the highest score — is what actually assigns labels; the boundary is just the seam where that rule is ambiguous. Because of this, multiclass plots can show several disconnected regions belonging to the same class, particularly when class distributions are complex or when one-vs-rest strategies are used.
Decision Boundaries in High-Dimensional Spaces
A two-dimensional chart of a decision boundary is only a direct, exact representation when the model genuinely uses just two features. The moment a model uses three, ten, or a thousand features, any 2D plot is necessarily a simplification.
Several common approaches make high-dimensional boundaries at least partially visible. Plotting two original features while holding all others fixed at a representative value (such as their mean) shows a "slice" through the full boundary — useful, but only accurate for that specific fixed setting of the other features. Creating multiple such slices across different feature pairs, similar in spirit to partial-dependence style analysis, builds a broader picture piece by piece.
Dimensionality-reduction techniques like Principal Component Analysis (PCA) project high-dimensional data down onto two or three new axes that capture the most variance; t-distributed Stochastic Neighbor Embedding (t-SNE) and Uniform Manifold Approximation and Projection (UMAP) instead try to preserve local neighborhood structure for visualization purposes [15]. All of these are genuinely useful for exploration, but none of them preserves the exact original geometry. A boundary that looks smooth and simple after a PCA projection may correspond to a far more complex boundary in the original feature space, and distances or apparent separations in a t-SNE or UMAP plot should not be read as exact distances in the original space at all — these techniques are explicitly designed to prioritize visual structure over strict geometric fidelity.
How to Visualize Decision Boundaries in Python
Complete scikit-learn Example
The following example generates a synthetic two-feature dataset, trains five contrasting classifiers, and plots each one's decision regions using scikit-learn's DecisionBoundaryDisplay [16].
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.inspection import DecisionBoundaryDisplay
from sklearn.metrics import accuracy_score
RANDOM_STATE = 42
# 1. Generate a synthetic two-feature, nonlinearly separable dataset
X, y = make_moons(n_samples=400, noise=0.25, random_state=RANDOM_STATE)
# 2. Split into training and test sets
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=RANDOM_STATE, stratify=y
)
# 3. Scale features (important for distance- and margin-based models)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
# 4. Define contrasting classifiers
classifiers = {
"Logistic Regression": LogisticRegression(random_state=RANDOM_STATE),
"Linear SVM": SVC(kernel="linear", random_state=RANDOM_STATE),
"RBF-Kernel SVM": SVC(kernel="rbf", gamma="scale", random_state=RANDOM_STATE),
"Decision Tree": DecisionTreeClassifier(max_depth=4, random_state=RANDOM_STATE),
"k-Nearest Neighbors": KNeighborsClassifier(n_neighbors=7),
}
# 5. Train, evaluate, and plot each model's decision boundary
fig, axes = plt.subplots(2, 3, figsize=(16, 10))
axes = axes.ravel()
for ax, (name, clf) in zip(axes, classifiers.items()):
clf.fit(X_train_scaled, y_train)
preds = clf.predict(X_test_scaled)
test_accuracy = accuracy_score(y_test, preds)
DecisionBoundaryDisplay.from_estimator(
clf,
X_train_scaled,
response_method="predict",
alpha=0.4,
ax=ax,
cmap="coolwarm",
)
ax.scatter(
X_train_scaled[:, 0], X_train_scaled[:, 1],
c=y_train, cmap="coolwarm", edgecolor="k", s=25
)
ax.set_title(f"{name}\nTest accuracy: {test_accuracy:.2f}")
ax.set_xlabel("Feature 1 (scaled)")
ax.set_ylabel("Feature 2 (scaled)")
axes[-1].axis("off")
plt.tight_layout()
plt.savefig("decision_boundaries.png", dpi=150)
plt.show()
This code does not fabricate output — running it will produce real accuracy values that depend on your environment's random state handling and library versions, so no specific numbers are claimed here.
How to Interpret the Plots
Each subplot shows the same underlying two-moons dataset, but a different boundary shape. Logistic regression and the linear SVM will show a single straight dividing line, which will systematically underfit the crescent-shaped classes. The RBF-kernel SVM should show a boundary that curves to follow the moon shapes far more closely. The decision tree will show a boundary made of stacked rectangular steps, and k-nearest neighbors will show a more organic, locally responsive boundary that can look either smooth or noisy depending on the chosen k.
Experiments to Try
Readers can change max_depth in the decision tree to see how a shallow tree underfits and a very deep tree starts drawing an overly intricate boundary around individual points. Changing n_neighbors in k-NN from 1 up to 50 shows the transition from a jagged, point-hugging boundary to an overly smooth one. Switching the SVM's gamma parameter between low and high values, and switching kernel between "linear" and "rbf", demonstrates directly how a linear boundary becomes a curved one. Crucially, test-set performance — not how attractive a boundary looks — should drive any real model choice; a highly contorted boundary can look impressive on a plot while performing worse than a simple one on genuinely unseen data.
Decision Boundaries and Overfitting
Boundary complexity sits directly at the center of the bias-variance trade-off. An overly simple boundary — a straight line forced onto genuinely curved class structure — underfits: it has high bias, misses real patterns, and produces both high training error and high validation error. A highly contorted boundary that wriggles around every individual training point can achieve near-zero training error while performing poorly on new data — a sign of high variance and overfitting, where the model has partly memorized noise rather than learned generalizable structure.
A useful boundary sits between these extremes: it captures genuine class separation while ignoring noise. Regularization is the primary tool for nudging a model away from the overfitting extreme, penalizing overly complex parameter values or overly deep trees. Visual smoothness alone is not proof of a good boundary — a smooth-looking boundary can still be poorly placed if the model's underlying assumptions do not match the true data structure, and a jagged boundary is not automatically worse if the true class structure genuinely is jagged (as can happen with certain fraud or anomaly patterns).
How to Evaluate Whether a Boundary Generalizes
The best-looking boundary in a training plot is not necessarily the best model — the correct way to know is disciplined evaluation on data the model never saw during training or tuning.
Proper practice separates data into training, validation, and test sets, or uses cross-validation to make more efficient use of limited data by rotating which portion serves as the held-out set. A confusion matrix breaks predictions down into true positives, true negatives, false positives, and false negatives, forming the basis for accuracy, precision, recall, and F1 score. ROC-AUC summarizes performance across all thresholds using the ROC curve; PR-AUC does the same using the precision-recall curve and is generally more informative under class imbalance. Log loss penalizes confident wrong predictions more heavily than uncertain ones, making it sensitive to calibration as well as accuracy. Calibration metrics specifically check whether stated probabilities match observed frequencies.
Situation | Useful metrics | Why they matter |
Balanced classes, no cost asymmetry | Accuracy, F1 score | Simple and representative when classes and costs are roughly equal |
Imbalanced classes | Precision-recall curve, PR-AUC | ROC-AUC can look misleadingly strong when negatives dominate |
Asymmetric costs (e.g., missed fraud vs. false alarm) | Precision, recall, cost-weighted metrics | Accuracy alone hides which type of error a model makes |
Need trustworthy probabilities | Log loss, calibration curves | Confirms scores can be used for downstream decisions, not just ranking |
Accuracy alone is rarely sufficient for imbalanced or high-cost classification — a model predicting "no fraud" for every transaction can post very high accuracy while catching zero actual fraud. Robustness checks, including performance across demographic or operational subgroups, add another layer of assurance that a boundary generalizes fairly and not just on average.
The Bayes Decision Boundary
The Bayes decision boundary is the theoretically optimal boundary achievable if the true class-conditional distributions and prior probabilities of each class were fully known [17]. It is defined using posterior probabilities — the probability of each class given the observed features — and the classifier that always picks the class with the highest posterior probability (assuming equal misclassification costs) achieves the lowest possible expected error, known as the Bayes error rate.
Real models never actually know the true distributions; they only estimate them from finite, noisy training data, so every real classifier is an approximation of the Bayes boundary, not the boundary itself. Even the Bayes-optimal classifier cannot achieve zero error whenever classes genuinely overlap in feature space — that overlap creates irreducible error, error no amount of additional data or better modeling can eliminate, because it reflects genuine ambiguity in the underlying data-generating process rather than a modeling shortcoming.
Real-World Applications
Medical screening. Features might include lab values, imaging measurements, and patient history; the boundary separates "likely disease" from "likely healthy." Moving the threshold toward higher sensitivity catches more true cases at the cost of more false alarms requiring follow-up testing — a trade-off clinicians and health systems weigh explicitly.
Fraud detection. Features often include transaction amount, location, timing, and account history. Because fraud is typically a small minority of transactions, threshold choice and precision-recall evaluation matter more here than in balanced problems.
Credit risk. Features include income, debt levels, and credit history. Moving the boundary changes who receives credit, connecting the mathematics directly to regulatory fairness and lending-policy concerns.
Spam filtering. Features include word frequencies, sender reputation, and message metadata. As shown earlier, threshold choice reflects a real trade-off between missed spam and wrongly blocked legitimate mail.
Manufacturing quality control. Sensor readings and imaging measurements feed a boundary separating "pass" from "defect," where threshold placement balances waste from false rejections against risk from missed defects.
Customer churn. Features like usage frequency, support tickets, and tenure feed a boundary separating "likely to churn" from "likely to stay," informing which customers receive retention outreach.
Image classification. High-dimensional pixel or learned-feature representations feed extremely complex boundaries, typically formed by neural networks, that cannot be visualized directly in the ways shown earlier in this guide.
None of these real systems is fully captured by a simple two-dimensional line — production models typically use dozens or hundreds of features, and the two-feature illustrations used for intuition throughout this guide are teaching tools, not full descriptions of deployed systems.
Common Misconceptions and Mistakes
A decision boundary is not always a straight line — only linear models produce straight-line boundaries; many algorithms produce curved, piecewise, or irregular ones. Not every classifier explicitly stores a geometric line or surface as an object; most models store parameters (weights, split rules, support vectors) from which a boundary can be derived, but the boundary itself is usually implicit rather than stored directly. A more complicated boundary is not always more accurate — overly complex boundaries frequently overfit and perform worse on new data than simpler ones. A point far from the boundary is not guaranteed to be correct, especially under distribution shift or poor calibration. The default 0.5 threshold is not always optimal — it is a mathematically convenient default that ignores the actual costs of different error types. A two-dimensional visualization does not show the complete high-dimensional model — it shows a slice or approximation. Decision boundaries do not apply to regression the same way they apply to classification, since regression predicts continuous values rather than discrete categories (though certain regression-based decision rules can define thresholds of their own). A perfectly separating training boundary does not guarantee good generalization — it can be a sign of overfitting rather than genuine skill. Probability calibration and threshold tuning are related but distinct tasks, as covered earlier. Finally, a boundary does not cause classes to exist; it merely describes the model's prediction rule given data that already reflects real underlying categories.
A Practical Workflow for Better Classification Decisions
Define the decision being automated and the real costs of each type of error before choosing any model.
Prepare and split the data correctly into training, validation, and test sets, guarding against leakage.
Establish a simple baseline model, such as logistic regression, before trying more complex alternatives.
Select appropriate features and preprocessing, including scaling for distance- and margin-based models.
Compare models with genuinely different inductive biases — linear, tree-based, and instance-based — rather than variations of the same family.
Tune hyperparameters using validation data or cross-validation, never the final test set.
Check calibration and thresholds explicitly, rather than defaulting to 0.5 without justification.
Evaluate subgroup performance and robustness, not just aggregate accuracy.
Investigate errors both near and far from the boundary to understand different failure modes.
Monitor for distribution shift after deployment, since a well-placed boundary can drift out of date as real-world data changes.
Putting the Concept Together
A decision boundary is not a separate object a model consults — it is the direct, visible consequence of a scoring function, a threshold, and the data used to learn both. Understanding it means understanding why logistic regression draws straight lines while kernel SVMs and neural networks draw curves, why moving a threshold can change outcomes without retraining anything, why a boundary that perfectly separates training data can still be a bad boundary, and why the "best" boundary is ultimately whichever one performs reliably on data the model has never seen. Every technique in this guide — from the linear equation wᵀx + b = 0 to Python's DecisionBoundaryDisplay to precision-recall curves — exists to help answer one question honestly: where should the line actually be drawn, and how do you know it is in the right place?
FAQ
What is a decision boundary in simple terms?
A decision boundary is the point, line, or surface where a machine learning model switches its prediction from one class to another. On one side, the model predicts one label; on the other side, it predicts a different label. It emerges from the model's learned scoring function rather than being drawn manually.
Is a decision boundary always a straight line?
No. Only linear models like plain logistic regression or a linear SVM produce straight-line (or flat hyperplane) boundaries. Kernel SVMs, decision trees, k-nearest neighbors, and neural networks can all produce curved, piecewise, or irregular boundaries depending on their structure and hyperparameters.
What is the difference between a decision boundary and a decision threshold?
A decision boundary is a geometric shape in feature space; a decision threshold is a numeric cutoff applied to a score or probability. For many classifiers, changing the threshold moves the effective boundary through feature space without retraining the underlying model at all.
What is a decision boundary in logistic regression?
In logistic regression, the boundary is where the predicted probability equals the chosen threshold, commonly 0.5. This corresponds exactly to the linear equation wᵀx + b = 0, since the sigmoid function that produces the probability equals 0.5 precisely when its input is zero.
How does an SVM choose its decision boundary?
A linear SVM chooses the hyperplane that maximizes the margin — the distance to the closest training points of each class. Kernel SVMs implicitly transform features into a higher-dimensional space, find a maximum-margin hyperplane there, and project it back as a curved boundary in the original feature space.
How do decision trees create boundaries?
Decision trees split feature space one variable at a time using threshold rules, producing boundaries that are axis-aligned and piecewise rectangular. Deeper trees can approximate more complex shapes but risk overfitting; shallower trees are more interpretable but may underfit curved true boundaries.
Can a decision boundary exist with more than two features?
Yes. With three features the boundary is typically a surface in three-dimensional space, and with many features it becomes a higher-dimensional object that cannot be drawn directly but still fully governs the model's predictions.
How do multiclass decision boundaries work?
With more than two classes, feature space divides into multiple decision regions. Boundaries occur where two or more class scores tie for the highest value, using strategies such as one-vs-rest, one-vs-one, or a unified multinomial model.
What makes a good decision boundary?
A good decision boundary captures genuine class structure in the data while ignoring noise, performs well on unseen validation and test data, and is placed with a threshold that reflects the real costs of different types of errors for the specific application.
How does overfitting affect a decision boundary?
Overfitting produces an overly contorted boundary that wraps tightly around individual training points, including noise, achieving very low training error but poor performance on new data. Regularization and simpler model choices help pull the boundary back toward genuine, generalizable structure.
Can changing the classification threshold move the boundary?
Yes. For classifiers that output a score or probability, adjusting the threshold shifts the resulting effective decision boundary through feature space, without any need to retrain the model's underlying parameters.
Why are predictions near the boundary uncertain?
Points near the boundary have scores very close to the decision threshold, meaning small amounts of noise, measurement error, or natural variation can flip the predicted label. This makes the boundary's immediate neighborhood inherently less certain than regions far from it.
How can I plot a decision boundary in Python?
Scikit-learn's DecisionBoundaryDisplay.from_estimator function plots a trained classifier's decision regions directly from two features, overlaying the training points for context. It works well for two-feature or two-feature-slice visualizations, as shown in the Python example in this guide.
Do regression models have decision boundaries?
Not in the same sense as classifiers, since regression predicts continuous numeric values rather than discrete categories. However, some regression-based systems apply their own thresholds to continuous outputs to make binary decisions, creating a threshold-like boundary in that specific downstream sense.
What is the Bayes decision boundary?
The Bayes decision boundary is the theoretically optimal boundary that would result from knowing the true class distributions and choosing the class with the highest posterior probability. It sets a lower bound on achievable error, and every real-world classifier is only an approximation of it, learned from finite, noisy data.
Key Takeaways
A decision boundary marks exactly where a classifier's predicted class changes across feature space.
The boundary is derived from a scoring function and a threshold, not stored as a separate manual object.
Linear models produce hyperplane boundaries; kernel methods, trees, k-NN, and neural networks can produce curved, piecewise, or irregular ones.
A decision boundary and a decision threshold are related but distinct — moving the threshold can shift the boundary without retraining.
Feature scaling, feature engineering, and data quality all directly shape where and how a boundary forms.
Overly complex boundaries risk overfitting; overly simple ones risk underfitting — evaluation on unseen data is the only reliable judge.
Two-dimensional visualizations and dimensionality-reduction projections are useful but do not represent the exact geometry of a high-dimensional boundary.
The Bayes decision boundary is the theoretical best case; every practical model is an approximation shaped by its data and assumptions.
Actionable Next Steps
Pick one dataset with two numeric features and plot several classifiers' boundaries side by side using the Python example provided.
Compare a linear model's boundary against a kernel SVM's boundary on the same nonlinear dataset to see the practical difference directly.
Experiment with a decision tree's max_depth parameter and observe how the boundary shifts from underfitting to overfitting.
Adjust a classification threshold away from 0.5 on a probability-based model and measure the resulting change in precision and recall.
Check whether your model's probabilities are calibrated using a calibration curve before trusting them for threshold-based decisions.
Evaluate any classifier using precision-recall curves, not just accuracy, whenever your classes are imbalanced.
Before deploying any classifier, document the real-world costs of false positives and false negatives to justify the chosen threshold.
Glossary
Bayes classifier — A classifier that assigns the class with the highest posterior probability, achieving the theoretically lowest possible error rate given known true distributions.
Bayes decision boundary — The optimal boundary produced by a Bayes classifier, representing the best achievable separation between classes.
Binary classification — A classification task involving exactly two possible classes.
Calibration — The degree to which a model's stated probabilities match observed real-world frequencies.
Class — One of the discrete categories a classifier assigns to input data.
Class imbalance — A situation where one class has far more examples than another in the training data.
Class probability — A model's estimated probability that a given input belongs to a particular class.
Classification threshold — The cutoff value applied to a score or probability to produce a final predicted label.
Decision boundary — The location in feature space where a classifier's predicted class changes.
Decision function — The underlying mathematical function that produces a model's score for a given input.
Decision region — A connected area of feature space assigned to a single predicted class.
Decision surface — A term often used interchangeably with decision boundary, especially in higher dimensions.
Feature — An individual measured or engineered variable used as model input.
Feature space — The full coordinate system defined by all the features a model uses.
Generalization — A model's ability to perform well on new, unseen data rather than only on its training data.
Hyperplane — A flat, one-dimension-lower object in feature space, such as a line in 2D or a plane in 3D.
Kernel trick — A technique that lets algorithms like SVMs compute similarity in a high-dimensional transformed space without explicitly constructing it.
Margin — The distance between a decision boundary and the nearest training points, central to how SVMs are trained.
Multiclass classification — A classification task involving three or more possible classes.
Nonlinear model — A model whose decision boundary cannot be described by a single linear equation.
Overfitting — When a model fits training data (including its noise) too closely, harming performance on new data.
Regularization — A technique that penalizes model complexity to reduce overfitting and stabilize the decision boundary.
Score — A raw numeric output from a model's decision function, before any threshold is applied.
Support vector — A training point closest to the decision boundary that directly influences an SVM's margin.
Underfitting — When a model is too simple to capture genuine structure in the data, producing high error on both training and new data.
Sources & References
Hastie, T., Tibshirani, R., & Friedman, J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed. Springer. n.d. https://hastie.su.domains/ElemStatLearn/
James, G., Witten, D., Hastie, T., & Tibshirani, R. An Introduction to Statistical Learning, 2nd ed. Springer. n.d. https://www.statlearning.com/
scikit-learn developers. "LogisticRegression." scikit-learn documentation. Accessed 2026-07-16. https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html
scikit-learn developers. "Multiclass and multioutput algorithms." scikit-learn documentation. Accessed 2026-07-16. https://scikit-learn.org/stable/modules/multiclass.html
scikit-learn developers. "Support Vector Machines." scikit-learn documentation. Accessed 2026-07-16. https://scikit-learn.org/stable/modules/svm.html
Cortes, C., & Vapnik, V. "Support-Vector Networks." Machine Learning, 20(3), 273–297. 1995.
scikit-learn developers. "Nearest Neighbors." scikit-learn documentation. Accessed 2026-07-16. https://scikit-learn.org/stable/modules/neighbors.html
scikit-learn developers. "Decision Trees." scikit-learn documentation. Accessed 2026-07-16. https://scikit-learn.org/stable/modules/tree.html
scikit-learn developers. "Ensemble methods." scikit-learn documentation. Accessed 2026-07-16. https://scikit-learn.org/stable/modules/ensemble.html
scikit-learn developers. "Linear and Quadratic Discriminant Analysis." scikit-learn documentation. Accessed 2026-07-16. https://scikit-learn.org/stable/modules/lda_qda.html
scikit-learn developers. "Naive Bayes." scikit-learn documentation. Accessed 2026-07-16. https://scikit-learn.org/stable/modules/naive_bayes.html
Goodfellow, I., Bengio, Y., & Courville, A. Deep Learning. MIT Press. 2016. https://www.deeplearningbook.org/
scikit-learn developers. "Probability calibration." scikit-learn documentation. Accessed 2026-07-16. https://scikit-learn.org/stable/modules/calibration.html
scikit-learn developers. "Precision, recall and F-measures." scikit-learn documentation. Accessed 2026-07-16. https://scikit-learn.org/stable/modules/model_evaluation.html
scikit-learn developers. "Manifold learning." scikit-learn documentation. Accessed 2026-07-16. https://scikit-learn.org/stable/modules/manifold.html
scikit-learn developers. "DecisionBoundaryDisplay." scikit-learn documentation. Accessed 2026-07-16. https://scikit-learn.org/stable/modules/generated/sklearn.inspection.DecisionBoundaryDisplay.html
Bishop, C. M. Pattern Recognition and Machine Learning. Springer. 2006.
Ng, A. "CS229 Lecture Notes: Supervised Learning, Discriminative Algorithms." Stanford University. n.d. https://cs229.stanford.edu/
scikit-learn developers. "Plot the decision boundaries of a VotingClassifier / Classifier comparison." scikit-learn examples gallery. Accessed 2026-07-16. https://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html
scikit-learn developers. "sklearn.datasets.make_moons." scikit-learn documentation. Accessed 2026-07-16. https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html
