What Is Label Shift?
- 2 days ago
- 33 min read
A hospital trains a diagnostic model when 2% of patients have a rare infection. Six months later, during an outbreak, 15% of patients test positive. The symptoms haven't changed. The disease still looks the same in the body. But the model's predictions quietly stop matching reality, and nobody flagged an error, because there wasn't one — the world just shifted underneath the model's assumptions.
TL;DR
Label shift means the class distribution P(Y) changes between training and deployment while the class-conditional feature distributions P(X∣Y) stay approximately the same ([Lipton et al., 2018][1]).
It is also called class-prior shift, prior probability shift, or, less precisely, target shift.
It differs from covariate shift (features change, P(Y∣X) stable) and concept shift (the X-to-Y relationship itself changes).
You can estimate the new class priors from unlabeled deployment data using methods like Black Box Shift Estimation (BBSE) ([Lipton et al., 2018][1]) or maximum-likelihood label-shift estimation (MLLS) ([Saerens et al., 2002][2]; [Garg et al., 2020][3]), then correct posterior probabilities with a simple Bayes-rule reweighting.
A mathematically correct label-shift correction applied to the wrong kind of shift can make a model worse, not better — so validation matters as much as the correction formula.
What Is Label Shift?
Label shift is a type of dataset shift in which the proportion of each class changes between training and deployment data, while the way each class produces its features stays the same. Also called class-prior shift, it lets teams correct a model's predictions using only the new class frequencies, without retraining, provided the stability assumption actually holds.
Table of Contents
What Label Shift Means
Label shift is a change in the class distribution P(Y) between the data a model was trained on and the data it now sees, while the class-conditional feature distribution P(X∣Y) — how each class generates its features — stays roughly the same.
In plain language: the mix of categories changes, but each category still "looks" the same. If disease prevalence doubles, sick patients still present the same symptom patterns they always did. What changes is how often you encounter a sick patient at all.
Formally, using the notation this article will keep throughout, let (P_s(X,Y)) denote the source (training) joint distribution and (P_t(X,Y)) denote the target (deployment) joint distribution. The classical label-shift assumption, first framed this way for black-box classifiers by Lipton, Wang, and Smola[1], states:
[ P_s(Y) \neq P_t(Y) \quad \text{while} \quad P_s(X\mid Y) = P_t(X\mid Y) ]
Here (P_s(Y)) and (P_t(Y)) are the class priors — the overall probability of each label — in the source and target domains. (P_s(X\mid Y)) and (P_t(X\mid Y)) are the class-conditional distributions: given that an example belongs to class y, how its features x are distributed. Label shift says these conditionals don't move; only the mixing weights between classes do.
An important and often-missed consequence: even though P(X∣Y) is stable, the marginal feature distribution P(X) and the posterior P(Y∣X) — the very quantity most classifiers try to estimate — can still change. If disease prevalence rises, then even for a patient with a fixed symptom profile, the probability that they actually have the disease rises too, because the base rate rose. This is the crux of why label shift causes real, measurable performance problems even when nothing about the underlying machine learning model or the disease biology changed.
The intuitive analogy: imagine a factory that always paints exactly 30% of its boxes red and 70% blue, and a machine trained to sort them by color under that mix. If the factory changes its production ratio to 60% red and 40% blue — without changing what "red" or "blue" paint looks like — the color-sorting logic is unaffected, but any downstream calculation that assumed the old 30/70 split (like a probability that a randomly-picked box is red) needs updating.
A realistic example: a bank's fraud-detection model is trained on a year of transactions where 0.4% are fraudulent. During a coordinated attack campaign, the fraud rate temporarily jumps to 2%. If the fraud patterns themselves (the way fraudulent transactions look, given that they are fraud) haven't changed, this is classical label shift, and the model's priors — not its underlying pattern recognition — need adjusting.
Label shift is also called class-prior shift or prior probability shift in the pattern-recognition and quantification literature, and sometimes target shift, though usage of "target shift" varies across papers and can occasionally include broader notions of change in the label-generating process. This article uses "label shift" and "class-prior shift" interchangeably, following the terminology in [Lipton et al. (2018)][1] and [Saerens et al. (2002)][2].
Source and Target Domains
Almost all supervised learning theory rests on one convenient fiction: that training data and future data are independent and identically distributed (i.i.d.) draws from the same distribution. In practice, this assumption is fragile. The source domain is the distribution the training set and validation set were drawn from. The target domain is the distribution the model actually encounters after deployment — next month's customers, next season's transactions, a different hospital's patient population.
The i.i.d. assumption fails for mundane reasons: data collection windows end, populations move, seasons change, marketing campaigns shift who visits a website, and adversaries adapt. Quiñonero-Candela, Sugiyama, Schwaighofer, and Lawrence[6], in the foundational edited volume on this subject, frame the broad problem as dataset shift: any situation where the joint distribution of inputs and outputs differs between training and test. Label shift is one specific, mathematically tractable slice of that broader problem.
Because label shift only changes the mixing proportions between classes — not what each class looks like — it is, in principle, one of the more correctable forms of shift. That correctability is exactly why it has attracted a dedicated body of estimation and correction methods, covered later in this article.
Real-World Examples
Each example below states the mechanism and the condition that would need to hold for it to be classical label shift, rather than assuming the label alone proves it.
Disease prevalence. A model trained on hospital data where a condition affects 3% of patients is deployed during an epidemic where prevalence rises to 12%. This qualifies as label shift only if the symptom-given-disease relationship is stable — the same underlying pathology, not a new variant with different presentation.
Fraud during an attack campaign. A bank sees a spike in fraud rate during a holiday shopping surge or a coordinated attack. This is label shift only if the fraudulent transactions during the surge resemble previously-seen fraud patterns; a genuinely new fraud technique would violate the stable-conditional assumption and start to look like concept drift instead.
Spam category proportions. An email provider's spam classifier was trained when phishing made up most spam; today, promotional spam dominates instead. If phishing emails still look like phishing emails (given that they are phishing), and only the category mix shifted, that portion of the shift is closer to label shift — though spam classification in practice often experiences label shift and concept shift simultaneously, as adversaries also change tactics.
Support-ticket categories by season. A software company's support-ticket classifier is trained on a normal month, then deployed during a product launch when "billing" tickets spike relative to "bug report" tickets. Stable ticket phrasing within each category, with only the category ratio changing, is the label-shift-consistent version of this scenario.
Wildlife prevalence across regions. An ecological image classification model trained on camera-trap images from one forest is deployed in a neighboring region where species proportions differ. Stability requires that a given species still looks the same across regions — lighting, camera angle, and subspecies variation can violate this.
Content-moderation category frequencies after a major event. A platform's moderation classifier, trained in ordinary conditions, is deployed during a breaking news event when the ratio of "misinformation" flags to "harassment" flags changes sharply. This is closer to label shift only if each category's underlying content patterns are unchanged; breaking events often also introduce genuinely new phrasing, which would violate the assumption.
None of these scenarios should be assumed to be pure label shift merely because class frequencies changed. A change in observed class ratios is necessary evidence, but not sufficient proof, that the label-shift assumption holds.
Label Shift vs. Other Types of Shift
Shift type | What changes | What stays stable | Example | Common response |
Label shift | P(Y), class priors | P(X∣Y), class-conditionals | Disease prevalence rises during an outbreak | Estimate new priors; reweight posteriors |
Covariate shift | P(X), feature distribution | P(Y∣X), posterior relationship | New customer demographic visits a site | Importance weighting on features; recalibration |
Concept shift (conditional shift) | P(Y∣X), the X→Y relationship itself | Often nothing convenient | Fraud tactics evolve to evade detection | Retraining with new labeled data |
General dataset shift | Any part of the joint P(X,Y) | Nothing guaranteed | Multiple factors change together | Case-by-case diagnosis before choosing a fix |
Real production systems rarely experience a single, pure shift type. A pandemic changes disease prevalence (label shift) and which symptoms get reported first because testing criteria change (a covariate shift in reported symptoms) and, over enough time, viral mutation can change the symptom-given-disease relationship itself (concept shift). Rabanser, Günnemann, and Lipton[5] found empirically that dimensionality-reduction-plus-two-sample-testing approaches can detect many kinds of shift, but detecting that something shifted is a different task from diagnosing which shift mechanism is responsible — a distinction this article returns to repeatedly.
Note also that class imbalance — many training sets simply have unequal class sizes by design or by nature — is not the same phenomenon as label shift. Class imbalance describes a property of a single dataset; label shift describes a change between two datasets or time periods. A severely imbalanced training set is a modeling challenge on its own (see SMOTE and stratified sampling as common responses); label shift is what happens when that imbalance itself moves over time. Similarly, label noise (incorrect labels), label bias (systematically wrong labels for a subgroup), sampling bias (non-representative collection), and data drift as a catch-all operational term for "something about the data pipeline changed" are all related but distinct ideas that should not be collapsed into "label shift" just because they involve labels or distributions.
Why Label Shift Matters
When class priors shift and a model's posterior estimates are not re-anchored to the new priors, several measurable consequences follow.
Accuracy can degrade even if the underlying discriminative signal (how features relate to labels within a class) hasn't changed at all, because the operating point of the classifier no longer matches reality.
Precision and recall move in opposite directions depending on the shift's direction: as a positive class becomes more prevalent, a threshold tuned to the old prior tends to under-flag it, hurting recall for the now-larger class.
Positive and negative predictive values are mathematically tied to the base rate through Bayes' theorem, so they are especially sensitive to unaddressed label shift, sometimes more so than sensitivity or specificity.
Calibration — whether a "70% probability" output actually happens 70% of the time — degrades directly, since calibration is inherently tied to the true base rate.
Threshold-dependent decisions, such as flagging a transaction for review or triggering a clinical alert, can become systematically miscalibrated to cost, over-triggering or under-triggering relative to what the organization intended.
Expected business or clinical cost changes because cost calculations usually multiply an error's probability by its consequence, and the probability term is exactly what label shift perturbs.
Resource allocation, such as staffing a fraud-review queue or a clinical unit, can be misjudged if it is driven by stale prevalence assumptions.
Model monitoring dashboards built around fixed thresholds and static baselines can flip from "green" to persistently "red," or the reverse, purely due to prior shift rather than any change in model quality.
Fairness and subgroup performance can be affected when class prevalence differs across demographic groups or operating environments; a model calibrated overall can still be poorly calibrated for a subgroup whose local prevalence diverges from the global one.
That fairness point deserves a caveat: prior correction alone is not a fairness fix. Adjusting for an aggregate change in class prevalence does not repair a model that encodes discriminatory feature-label relationships, nor does it address the case where P(X∣Y) genuinely differs across subgroups. Label-shift correction is a narrow statistical tool, not a general remedy for equitable outcomes.
The Math: Deriving Posterior Correction
Bayes' theorem states that the posterior probability of a class given features is proportional to the likelihood of those features under that class, times the class's prior probability:
[ P(y \mid x) \propto P(x \mid y), P(y) ]
Under the label-shift assumption, (P_s(x\mid y) = P_t(x\mid y)) for every class y — only the prior term changes between domains. This lets us relate the target posterior to the source posterior, which a model has already learned, through a simple ratio of priors.
Starting from Bayes' rule in each domain:
[ P_s(y\mid x) = \frac{P_s(x\mid y) P_s(y)}{\sum_{y'} P_s(x\mid y') P_s(y')}, \qquad P_t(y\mid x) = \frac{P_t(x\mid y) P_t(y)}{\sum_{y'} P_t(x\mid y') P_t(y')} ]
Because (P_s(x\mid y) = P_t(x\mid y)), we can substitute the (already-learned) source likelihood into the target expression, and after algebraic rearrangement (dividing both numerator and denominator by (P_s(x\mid y)) after expressing it via the source posterior), we arrive at the central posterior-adjustment formula, consistent with the classical result in [Saerens, Latinne, and Decaestecker (2002)][2] and the black-box formulation in [Lipton, Wang, and Smola (2018)][1]:
[ P_t(y\mid x) = \frac{P_s(y\mid x), \dfrac{P_t(y)}{P_s(y)}}{\sum_{y'} P_s(y'\mid x), \dfrac{P_t(y')}{P_s(y')}} ]
Reading this in words: take the model's original posterior for class y, multiply it by how much more (or less) common class y has become, then divide by the sum of that same operation applied to every possible class, so the results add up to 1 again.
Define the importance weight:
[ w(y) = \frac{P_t(y)}{P_s(y)} ]
This is simply the factor by which class y's prevalence has changed. A class that doubled in frequency gets (w(y)=2); a class that halved gets (w(y)=0.5). The correction formula reweights each class's posterior contribution by its own (w(y)), then renormalizes.
A few essential points follow directly from the derivation:
Why the prior ratio appears. The ratio (w(y)) is exactly the multiplicative adjustment needed to convert a probability that was implicitly conditioned on the old mixing proportions into one conditioned on the new mixing proportions, because Bayes' rule ties posteriors to priors through simple multiplication.
Why normalization is required. Multiplying each class's posterior by a different weight breaks the constraint that probabilities across classes sum to 1. Dividing by the sum of all weighted terms restores a valid probability distribution — this is the same normalizing role the denominator always plays in Bayes' rule.
Why the correction is multiclass-compatible. Nothing in the derivation assumed only two classes; the sum in the denominator runs over however many classes exist, so the formula generalizes cleanly beyond binary classification.
What happens when a source prior is extremely small or zero. If (P_s(y)) is very small, (w(y)) becomes very large and numerically unstable; if (P_s(y)=0) — meaning the class never appeared in the source data at all — the weight is undefined, and no amount of correction can recover information the model never had a chance to learn. This is an identifiability problem, not merely a numerical one, discussed further in the failure-modes section.
How thresholds may need to change. A decision rule like "flag if P(y=1∣x) > 0.5" was implicitly tuned to the old base rate. After correction, the same threshold may no longer represent the same cost trade-off, so threshold-dependent business logic often needs re-examination alongside the posterior correction itself.
Worked Numerical Example
Consider a binary logistic-regression-style fraud classifier.
1. Source class priors: (P_s(\text{fraud}) = 0.02), (P_s(\text{legit}) = 0.98).
2. Target class priors: during an attack campaign, (P_t(\text{fraud}) = 0.08), (P_t(\text{legit}) = 0.92).
3. Model's source-domain posterior for one transaction: for a specific transaction x, the trained model outputs (P_s(\text{fraud}\mid x) = 0.30) and (P_s(\text{legit}\mid x) = 0.70).
4. Prior ratios (importance weights):
[ w(\text{fraud}) = \frac{0.08}{0.02} = 4.0, \qquad w(\text{legit}) = \frac{0.92}{0.98} \approx 0.9388 ]
5. Unnormalized adjusted values:
[ 0.30 \times 4.0 = 1.20, \qquad 0.70 \times 0.9388 \approx 0.6571 ]
Class | Source posterior | Weight w(y) | Unnormalized product |
Fraud | 0.30 | 4.0000 | 1.2000 |
Legit | 0.70 | 0.9388 | 0.6571 |
6. Normalization step: the sum of the unnormalized values is (1.2000 + 0.6571 = 1.8571).
7. Final target-domain posterior:
[ P_t(\text{fraud}\mid x) = \frac{1.2000}{1.8571} \approx 0.6462, \qquad P_t(\text{legit}\mid x) = \frac{0.6571}{1.8571} \approx 0.3538 ]
8. Practical interpretation: for this particular transaction, the raw model output suggested a 30% chance of fraud — likely below a typical review threshold. After correcting for the fact that fraud has become four times more common in the deployment window, the same feature pattern corresponds to roughly a 65% chance of fraud, comfortably above most review thresholds. Nothing about the model's understanding of the transaction's features changed; only the base-rate context did. This demonstration assumes the model's original posterior estimate and the class-conditional stability were adequate; if the classifier's underlying probability estimates were themselves poorly calibrated, this correction would faithfully reweight a flawed input and could still produce a misleading output, a caveat covered further in the estimation and failure-mode sections.
Detecting Label Shift
Detecting that something changed and establishing that the change is label shift specifically are two different tasks, and conflating them is a common analytical mistake.
Monitoring predicted-label frequencies. The simplest signal: track the proportion of each predicted class over time. A change here is consistent with label shift but can equally reflect covariate shift, concept shift, or a broken data pipeline masquerading as a statistical shift.
Monitoring predicted-probability distributions. Beyond hard labels, tracking the full distribution of predicted probabilities (not just the argmax) can reveal subtler shifts and is more sensitive to changes in calibration.
Confusion-matrix-based methods. If a labeled reference sample and its confusion matrix are known, comparing predicted-label frequencies against what the confusion matrix would predict under the old priors offers a principled signal for whether priors have moved.
Black Box Shift Detection (BBSD). Introduced alongside BBSE by [Lipton, Wang, and Smola (2018)][1], BBSD's key insight is that if a classifier's confusion matrix is invertible, then testing whether the predicted-label distribution has changed is, under label shift, mathematically equivalent to testing whether the true-label distribution has changed — without needing target-domain labels at all. This turns a hard-to-observe test (has P(Y) changed?) into an observable one (has the classifier's output distribution changed?).
Two-sample tests. Statistical tests (such as tests on predicted-class histograms, or dimensionality-reduced feature representations) compare a reference window against a current window. [Rabanser, Günnemann, and Lipton (2019)][5] empirically evaluated a range of dimensionality-reduction techniques combined with two-sample testing and found this combination performed strongly across a broad set of shift types they studied, though "performs well at detecting shift" and "correctly identifies the shift mechanism" remain distinct claims.
Small labeled target samples and delayed ground-truth labels. Where feasible, periodically labeling a small target sample gives the most direct evidence, though ground-truth labels (a confirmed diagnosis, a confirmed fraud chargeback) often arrive weeks or months after the prediction, creating an unavoidable monitoring lag.
Statistical power and minimum sample size. Small monitoring windows have low statistical power to detect real shifts and are simultaneously prone to false alarms from ordinary sampling noise; both failure directions need to be weighed when setting window sizes.
Multiple-testing concerns. Running many per-segment or per-feature shift tests simultaneously inflates the chance of a false positive somewhere; correction procedures or holistic tests are advisable when monitoring many slices at once.
Dashboards and alert thresholds. Automated dashboards are useful for triage but are not a substitute for the qualitative judgment involved in confirming a shift's mechanism; a threshold-crossing alert should be treated as the start of an investigation, not its conclusion.
Crucially, changes in P(X) can themselves arise from label shift — if the classes have different feature distributions and their proportions change, the overall feature distribution shifts as a mathematical consequence, even though nothing about P(X∣Y) moved. This means a feature-distribution alert alone cannot distinguish label shift from covariate shift; further diagnosis is required.
Estimating Target Class Priors
Once a plausible label shift is suspected, the practical goal becomes estimating (P_t(Y)) using labeled source data plus unlabeled target data — since target labels are, by definition, what's usually missing.
Confusion-matrix / moment-matching estimation (BBSE). [Lipton, Wang, and Smola (2018)][1] proposed Black Box Shift Estimation: use a black-box classifier's confusion matrix (estimated on labeled source/held-out data) together with the observed distribution of predicted labels on unlabeled target data, and solve a linear system to back out the implied target class priors. It requires an invertible confusion matrix and works even with imperfect, uncalibrated classifiers, provided that invertibility condition holds and better predictors yield tighter estimates.
Maximum-likelihood label-shift estimation (MLLS). Rooted in the iterative procedure of [Saerens, Latinne, and Decaestecker (2002)][2], MLLS treats the target priors as parameters to be found by maximizing the likelihood of the observed (unlabeled) target features under the model, typically via an expectation-maximization (EM) loop that alternates between estimating priors and re-adjusting posteriors until convergence. [Garg, Wu, Balakrishnan, and Lipton (2020)][3] provided the first rigorous theoretical characterization of MLLS and showed it is roughly equivalent to BBSE under a particular choice of calibration, unifying what had looked like two separate approaches.
Soft versus hard classifier outputs. BBSE's original formulation can use either hard predicted labels (a confusion matrix of counts) or soft posterior probabilities; MLLS inherently works with soft posteriors. Soft-output methods generally use more information per example but are more sensitive to calibration quality.
The role of calibration. [Garg et al. (2020)][3] identified classifier calibration as a key consistency condition for MLLS; a poorly calibrated classifier (common in modern high-capacity models) can bias the estimated priors even when the label-shift assumption technically holds. [Alexandari, Kundaje, and Shrikumar (2020)][7] built directly on this, showing that combining MLLS with a "bias-corrected calibration" step outperformed both BBSE and the regularized approach below across the datasets and shifts they tested, and that the maximum-likelihood objective is provably concave — useful because it guarantees the EM procedure won't get stuck in a bad local optimum.
Regularized label-shift estimation (RLLS). [Azizzadenesheli, Liu, Yang, and Anandkumar (2019)][4] proposed Regularized Learning under Label Shifts, which estimates importance weights and derives a generalization bound for the resulting classifier that depends on function-class complexity rather than raw data dimensionality — useful because it explicitly accounts for uncertainty in the estimated weights, which matters most in the small-sample regime where naive estimates are noisy.
Small-sample stabilization. All estimation methods become noisier as the unlabeled target sample shrinks; regularization (as in RLLS), shrinkage toward the source prior, or reporting confidence intervals alongside point estimates are standard mitigations.
Simple estimation from a labeled target sample. When a labeled target sample is available, even a modest one, simply computing empirical class frequencies directly is often the most robust option, since it makes no assumption about classifier calibration or confusion-matrix invertibility at all — though it requires the very labels that are frequently scarce or delayed.
Method | Basic idea | Key inputs | Main assumption | Best suited for |
BBSE | Confusion matrix + moment matching | Held-out confusion matrix, unlabeled target predictions | Invertible confusion matrix | Robust estimation with imperfect/uncalibrated classifiers |
MLLS / EM | Maximize likelihood of target features via iterative reweighting | Source-trained posteriors, unlabeled target data | Classifier calibration; convergence of EM | Cases with a reasonably calibrated base classifier |
RLLS | Regularized importance-weight estimation with generalization bound | Labeled source, unlabeled target | Bounded weight variance; small-sample regime | Small target samples, need for theoretical guarantees |
Bias-corrected MLLS | MLLS plus explicit recalibration step | Same as MLLS, plus a calibration set | Calibration can be learned/corrected | Deep-learning classifiers with known miscalibration |
Direct labeled-target counting | Empirical class frequency on a labeled sample | A labeled target sample | None beyond standard sampling assumptions | When even a small labeled sample is obtainable |
No single method here is universally best; the strongest empirical results in [Alexandari et al. (2020)][7] favored calibrated MLLS on their benchmarks, but BBSE's robustness to poor calibration and RLLS's explicit uncertainty accounting make them preferable in other regimes, particularly when a classifier's calibration is unknown or unverifiable.
Correcting a Model Under Label Shift
Several distinct correction strategies exist, and they are not interchangeable.
Posterior probability adjustment (post-hoc, no retraining). Apply the Bayes-rule reweighting formula derived earlier directly to an already-trained model's outputs. This is the fastest and cheapest option and is exactly what BBSE/MLLS-style prior estimation feeds into.
Decision-threshold adjustment. Rather than (or in addition to) reweighting probabilities, adjust the decision threshold to reflect the new base rate and cost structure — useful when a full posterior recalibration is impractical but a single operating point needs to move.
Importance weighting for reweighted empirical risk. Instead of correcting outputs after the fact, reweight training examples by (w(y)) and retrain (or fine-tune) so the model directly optimizes for the target distribution; this is the basis of the RLLS approach in [Azizzadenesheli et al. (2019)][4].
Recalibration. Independent of the prior-shift correction, recalibrating a classifier's probability outputs (e.g., with a held-out calibration set) improves the reliability of the inputs to any prior-correction formula, since the derivation assumes the source posterior is a faithful representation of P_s(y∣x).
Retraining with representative target data. When enough labeled target data becomes available, retraining directly on it sidesteps the label-shift correction machinery altogether and is the most robust option — at the cost of data collection and engineering effort.
Combining prior correction with human review. For high-stakes decisions (clinical, financial, legal), an automated prior correction can be paired with routing borderline or high-impact cases to human reviewers, rather than fully automating the adjusted decision.
Cases where correction should not be applied. If diagnostic evidence points toward covariate shift or concept shift rather than label shift, applying the label-shift correction formula is not merely unhelpful — it actively distorts otherwise-valid posteriors, because it assumes a stability condition (P(X∣Y) unchanged) that doesn't hold.
It helps to keep four different actions conceptually separate: post-hoc correction without retraining (reweighting a fixed model's outputs), reweighting source samples and retraining (changing the training objective, not just its outputs), fine-tuning on labeled target data (using new information the original training set lacked), and rebuilding the model because the class-conditionals changed (an admission that the label-shift assumption itself has failed, and a different problem needs solving).
Implementation Workflow
Define source and target windows. Fix the exact time ranges, data slices, or populations being compared, and document them.
Validate label and feature semantics. Confirm that a "positive" label means the same thing in both windows and that feature-generation logic (sensors, forms, extraction pipelines) hasn't silently changed.
Establish baseline priors and confusion matrices. Compute (P_s(Y)) and the classifier's confusion matrix on held-out source data.
Collect a target sample. Gather unlabeled (and, if possible, a small labeled) sample from the deployment window.
Test for meaningful change. Run detection methods (BBSD, two-sample tests) to establish whether a statistically and operationally meaningful shift exists at all.
Assess whether the label-shift assumption is plausible. Combine domain knowledge with diagnostic checks (subgroup analysis, feature-distribution comparisons within predicted classes) before assuming label shift specifically.
Estimate target priors. Apply BBSE, MLLS/EM, or RLLS as appropriate, ideally cross-checking more than one method.
Inspect uncertainty and stability. Examine confidence intervals or bootstrap intervals around the estimated priors; treat wide or unstable intervals as a signal to gather more data before acting.
Apply correction in shadow mode. Run the corrected model alongside the production model without affecting live decisions, logging both outputs.
Evaluate with target labels when available. As delayed ground truth arrives, compare corrected versus uncorrected performance directly.
Deploy with safeguards. Roll out gradually, with fallback logic if the correction behaves unexpectedly, and clear rollback criteria.
Continue monitoring. Treat this as an ongoing cycle, not a one-time fix, since priors can continue to move.
Artifacts worth logging at each stage include: predicted probabilities, predicted classes, model version, data slice identifiers, timestamps, source priors, estimated target priors, the confusion matrix used, calibration metrics, the correction weights applied, confidence or bootstrap intervals around estimates, and downstream decision outcomes. This audit trail is what makes it possible to diagnose, months later, whether a performance change came from label shift, from a model update, or from a pipeline bug.
Python Example: Prior-Based Posterior Adjustment
The following is an educational implementation, not a production-ready library. It performs the multiclass posterior correction derived earlier, assuming source and target priors are already estimated.
import numpy as np
def adjust_posteriors_for_label_shift(source_posteriors, source_priors, target_priors):
"""
Adjust a classifier's source-domain posteriors for label shift.
source_posteriors: array of shape (n_samples, n_classes),
each row a probability distribution over classes from the
original (source-trained) model.
source_priors: array of shape (n_classes,), P_s(y) for each class.
target_priors: array of shape (n_classes,), estimated P_t(y).
Returns: array of shape (n_samples, n_classes), corrected
target-domain posteriors, each row normalized to sum to 1.
Educational implementation only -- not a complete production system.
"""
source_posteriors = np.asarray(source_posteriors, dtype=float)
source_priors = np.asarray(source_priors, dtype=float)
target_priors = np.asarray(target_priors, dtype=float)
n_classes = source_posteriors.shape[1]
if source_priors.shape[0] != n_classes or target_priors.shape[0] != n_classes:
raise ValueError("Dimension mismatch between posteriors and priors.")
if np.any(source_priors <= 0):
raise ValueError(
"Source priors must be strictly positive; a zero or negative "
"source prior makes the importance weight undefined or invalid."
)
if np.any(target_priors < 0) or not np.isclose(target_priors.sum(), 1.0, atol=1e-6):
raise ValueError("Target priors must be non-negative and sum to 1.")
# Importance weight w(y) = P_t(y) / P_s(y) for each class
weights = target_priors / source_priors
# Reweight each class's posterior contribution
unnormalized = source_posteriors * weights # broadcasts over rows
# Normalize each row so probabilities sum to 1 again
row_sums = unnormalized.sum(axis=1, keepdims=True)
if np.any(row_sums <= 0):
raise ValueError("Encountered a row with zero total weight; check inputs.")
corrected_posteriors = unnormalized / row_sums
return corrected_posteriorsThis function checks dimensions, rejects invalid or zero source priors, and normalizes its output — but it does not estimate target priors itself (that requires BBSE, MLLS, or a labeled sample, as described above), nor does it handle calibration, confidence intervals, or logging, all of which a production system would need.
Evaluating Whether Correction Helped
Evaluation should use labeled target data whenever it becomes available, and should compare several models side by side rather than judging the corrected model in isolation.
Log loss and Brier score directly penalize miscalibrated probabilities and are often more sensitive to label-shift effects than accuracy alone.
Expected calibration error (ECE) measures the gap between predicted confidence and observed accuracy, though its own estimator has known limitations — binning choices can meaningfully change the reported number, so it should be interpreted cautiously rather than as a single definitive figure.
Accuracy and balanced accuracy (which averages per-class recall) — balanced accuracy is often more informative under prior shift, since raw accuracy can be inflated or deflated purely by base-rate changes.
Precision, recall, and F1 should be examined per class, since label shift affects classes asymmetrically.
Area under the ROC curve (AUC-ROC), where appropriate, is relatively insensitive to prior shift itself (since it depends on the ranking of scores, not the base rate), making it a useful check on whether the model's underlying discrimination — as opposed to its calibration — has changed at all.
Area under the precision–recall curve (AUC-PR), where appropriate, is more sensitive to prior shift than AUC-ROC and is often more informative for rare-positive-class problems like fraud or disease detection.
Calibration plots (reliability diagrams) visualize whether predicted probabilities match observed frequencies across probability bins.
Decision- or cost-sensitive metrics, reflecting the actual business or clinical cost of false positives versus false negatives, translate statistical improvement into a decision-relevant one.
Class-prior estimation error, when the true target priors are eventually known (e.g., from delayed labels), directly measures how well BBSE, MLLS, or RLLS performed at their core task.
Accuracy alone can conceal important effects: a model can show stable or even improved accuracy after a prior shift while its calibration, and the reliability of any probability-based decision threshold, has quietly deteriorated. The most informative comparisons place the uncorrected model, a prior-corrected model, a recalibrated model, a retrained or fine-tuned model, and a simple baseline (such as always predicting the new majority class) side by side on the same held-out labeled sample.
Assumptions and Failure Modes
Label-shift correction rests on assumptions that frequently do not hold perfectly in practice, and understanding where it breaks matters as much as understanding the formula.
Violation of P_s(X∣Y) = P_t(X∣Y). If class-conditionals genuinely changed — new symptom variants, evolved fraud tactics — the correction formula is being applied outside its valid domain, and it can produce confidently wrong probabilities.
New classes. A target class with positive probability but zero representation in the source data is fundamentally unidentifiable through prior correction; the model never learned what that class looks like at all.
Missing source support. Related to new classes: even a rare but nonzero source class can produce unstable, high-variance weight estimates.
Label-definition changes. If what counts as "fraud" or "positive" was redefined (a policy change, a new diagnostic criterion), any apparent shift may reflect a definitional change rather than a real distributional one.
Conditional shift within a class. Subtler than a wholesale change, this occurs when a class's internal composition shifts (different fraud sub-types becoming more common within "fraud" overall) even while the class-level label proportions look stable.
Poor calibration. As [Garg et al. (2020)][3] and [Alexandari et al. (2020)][7] emphasize, an uncalibrated classifier undermines the assumptions behind MLLS specifically, and biases the resulting prior estimates.
Nearly singular or ill-conditioned confusion matrices. BBSE's linear-system solution becomes numerically unstable when the confusion matrix is close to non-invertible, which tends to happen with weak classifiers or highly correlated classes.
Weak classifiers. A classifier barely better than random provides little separating signal for BBSE or MLLS to exploit, regardless of the estimation method chosen.
Severe class imbalance compounds most of the above issues, since rare classes contribute the least stable estimates.
Very small target samples inflate the variance of any prior estimate, however it's computed.
Delayed or biased target labels. If the labels that do arrive are systematically different from the labels that will eventually arrive for the full population (e.g., only confirmed fraud cases get labeled quickly, while false negatives take longer to surface), naive evaluation can be misleading.
Selection bias in which examples get monitored, labeled, or reviewed can distort both detection and correction.
Feedback loops. A deployed model's own decisions (denying a loan, flagging a transaction) can change the population that gets observed going forward, creating a shift the model itself induced.
Nonstationary or rapidly changing target priors, where the "target distribution" is itself a moving target rather than a fixed new state, strain any static correction approach.
Subgroup-specific shifts can be masked by an aggregate correction that looks fine on average but is wrong for a specific segment.
Data-pipeline errors masquerading as statistical shift. A broken upstream feature, a schema change, or a logging bug can produce a class-distribution change that looks statistically identical to genuine label shift but has an entirely different fix.
Overcorrection occurs when an estimation method overreacts to noisy target data, especially with small samples, pushing corrected posteriors further from the truth than the uncorrected ones were.
Extreme or unstable importance weights — very large (w(y)) values from very rare source classes — can dominate the correction and destabilize otherwise-reasonable posteriors.
Practical safeguards include regularization of estimated weights (as in RLLS), clipping only when there is a principled justification tied to known plausible prior ranges, reporting confidence intervals rather than bare point estimates, running corrections in shadow mode before full deployment, defining fallback behavior for cases where the correction produces implausible outputs, and scheduling periodic labeled audits rather than relying solely on the correction's own internal signals. A mathematically valid correction applied under the wrong assumption — genuine covariate or concept shift misdiagnosed as label shift — can measurably make performance worse, not better, which is why diagnosis should precede correction rather than follow automatically from any detected change.
Production Monitoring
Sustained monitoring, not a one-time correction, is what makes label-shift handling durable in production MLOps practice.
Fixed versus rolling reference windows. A fixed baseline detects drift from a known-good period; a rolling window adapts to gradual change but can mask slow drift if it moves along with it.
Seasonality. Many class-prior changes are cyclical (holiday fraud spikes, seasonal illness patterns) and should be modeled as expected variation rather than triggering unnecessary alarms every cycle.
Alert thresholds should be set with both statistical and operational significance in mind — a statistically detectable shift that changes decisions by a negligible amount may not warrant action.
Confidence intervals around monitored quantities help distinguish genuine movement from sampling noise, especially in low-traffic segments.
Repeated testing across many time windows or segments should account for multiple-comparisons effects, or false alarms will accumulate.
Retraining triggers should be defined in advance — a specific magnitude or duration of shift that prompts a retraining or recalibration cycle, rather than ad hoc decisions each time.
Human review processes for borderline or high-consequence cases provide a safety net beyond automated correction.
Delayed labels should be incorporated into monitoring as soon as they arrive, even if that means revisiting decisions made weeks earlier.
Segmented monitoring by subgroup, region, or channel catches shifts that would be invisible in an aggregate view.
Model-version changes should be logged alongside distributional monitoring, since a performance change coinciding with a model deployment may have nothing to do with label shift at all.
Data-contract checks (schema validation, expected value ranges) catch pipeline errors before they are mistaken for statistical shift.
Rollback plans should specify exactly how to revert a correction or a model version if monitoring reveals it was a mistake.
Production checklist: confirm the data pipeline and label definitions are unchanged; compare current predicted-class frequencies against the seasonal baseline; run a two-sample or BBSD-style test for statistical significance; check whether the change is plausibly seasonal; if not, estimate updated priors via at least two independent methods; inspect confidence intervals on the new estimates; deploy any correction in shadow mode first; compare corrected, uncorrected, and baseline performance once labels arrive; document the decision and update the reference window; schedule the next review.
Advanced Topics
Online or time-varying label shift. Rather than a single discrete shift from one static distribution to another, priors can drift continuously. Recent theoretical work, including online label-shift formulations building on the estimation methods above, studies dynamic-regret guarantees for continuously updating prior estimates rather than treating each deployment window as a fresh, isolated estimation problem.
Generalized label shift. Extensions of the classical assumption relax exact equality of class-conditionals to approximate or representation-level invariance — the idea that P(X∣Y) need only be stable within some learned representation space, not in raw feature space, which connects label-shift theory to representation-learning research.
Mixtures of label and covariate shift. Real deployments rarely present pure label shift; more often, both class priors and within-class feature patterns move simultaneously, requiring combined diagnostic and correction strategies rather than a single formula.
Representation-dependent assumptions. Because deep models often operate on learned embeddings rather than raw features, whether the label-shift assumption "holds" can depend on which representation layer is being examined — an assumption might be reasonable at one layer of a network and violated at another.
Calibration under label shift. [Alexandari, Kundaje, and Shrikumar (2020)][7] demonstrated that calibration quality and label-shift correction interact tightly; ongoing research into calibration methods specifically designed to remain robust under prior shift builds directly on this line of work.
Label-shift estimation with limited labeled target data. Methods like RLLS [Azizzadenesheli et al. (2019)][4] were explicitly designed with the low-labeled-sample regime in mind, since real deployments rarely enjoy abundant fresh labels.
Identifiability versus practical reliability. A method can be theoretically consistent — guaranteed to converge to the correct answer given infinite data and a perfectly satisfied assumption — while still performing poorly in a specific, finite, imperfect real deployment. Practitioners should treat consistency proofs as a floor, not a guarantee of real-world performance.
Decision Guide: Is This Likely to Be Label Shift?
This is a reasoning aid, not a formal proof procedure.
Did only the observed class proportions appear to change, with feature patterns within each class looking stable on inspection?
Are label definitions and collection criteria identical across the two periods being compared?
When you examine features within a single predicted class, do they look similar across the source and target windows?
Are any target classes entirely absent from the source data (a strong signal that correction alone cannot fix the problem)?
Do class-conditional diagnostics — comparing feature distributions within each class across time — show meaningful divergence?
Is any target-label evidence available, even a small delayed or spot-checked sample, to directly verify the new class proportions?
Would applying a prior correction be testable in shadow mode before it affects real decisions?
If most answers point toward "yes, proportions changed but patterns within classes look stable, and I can verify this in shadow mode," proceeding with the estimation-and-correction workflow above is reasonable. If several answers raise doubts — new classes, uncertain label definitions, no way to verify — treating the situation as label shift prematurely risks the overcorrection failure mode described earlier.
FAQ
What is label shift in simple terms?
Label shift is when the mix of categories in your data changes over time or between environments — for example, more of your customers now belong to one category than before — while each category still looks the same as it always did. The categories didn't change; how often you see each one did.
What is another name for label shift?
Label shift is also called class-prior shift or prior probability shift, terms used more or less interchangeably in the statistics and machine-learning literature going back to [Saerens, Latinne, and Decaestecker (2002)][2]. "Target shift" is sometimes used as well, though that term is applied less consistently across different papers.
What is the difference between label shift and covariate shift?
Under label shift, the class proportions P(Y) change while the way each class produces features, P(X∣Y), stays the same. Under covariate shift, it's the reverse: the feature distribution P(X) changes while the relationship between features and labels, P(Y∣X), is assumed stable. They call for different diagnostic checks and different correction techniques, and conflating them can lead to applying the wrong fix.
Is label shift the same as concept drift?
No. Concept drift (or concept shift) refers to a change in the underlying relationship between features and labels itself — P(Y∣X) genuinely changing, such as fraud patterns evolving to look different. Label shift assumes that relationship-given-class is stable and only the class mix has moved. In practice, real systems can experience both at once, which is why diagnosis should precede correction.
Is class imbalance a form of label shift?
Not by itself. Class imbalance describes an unequal class distribution within a single dataset, which is a modeling challenge addressed by techniques like SMOTE or stratified sampling. Label shift specifically describes a change in class proportions between two datasets or time periods. A dataset can be imbalanced without ever experiencing label shift, and vice versa.
How can label shift be detected without target labels?
Black Box Shift Detection, introduced by [Lipton, Wang, and Smola (2018)][1], exploits the fact that under label shift, a change in the true label distribution produces a detectable change in the predicted-label distribution, provided the classifier's confusion matrix is invertible. Two-sample statistical tests on predicted-label or predicted-probability distributions offer a complementary, model-agnostic detection route, as studied empirically by [Rabanser, Günnemann, and Lipton (2019)][5].
What is Black Box Shift Estimation?
Black Box Shift Estimation (BBSE) is a method from [Lipton, Wang, and Smola (2018)][1] that estimates the new target class priors using an arbitrary pre-trained "black box" classifier's confusion matrix together with its predictions on unlabeled target data, solved as a linear system. It works even when the underlying classifier is imperfect or poorly calibrated, as long as its confusion matrix is invertible.
What role does the confusion matrix play?
The confusion matrix, estimated on labeled held-out source data, encodes how often the classifier predicts each label given each true label. BBSE uses this matrix to translate an observed shift in predicted-label frequencies on target data back into an estimate of the shift in true-label frequencies, which is the quantity actually needed for correction.
Why does calibration matter?
Maximum-likelihood label-shift estimation assumes the classifier's posterior probabilities are calibrated — that a predicted 70% really does correspond to a 70% empirical rate. [Garg et al. (2020)][3] identified calibration as a formal consistency condition for MLLS, and [Alexandari, Kundaje, and Shrikumar (2020)][7] showed that adding an explicit bias-correction calibration step substantially improved MLLS's real-world performance compared to using raw, uncalibrated model outputs.
Can label shift be fixed without retraining?
Yes, in many cases. Posterior probability adjustment applies the Bayes-rule reweighting formula directly to an existing model's outputs, requiring only estimated target priors rather than a new training run. This is usually the fastest and cheapest first response, though it depends on the label-shift assumption actually holding and on the quality of the estimated priors.
When does label-shift correction fail?
Correction can fail or actively harm performance when the class-conditional stability assumption is violated (genuine concept shift is present), when a target class has no representation in the source data, when the classifier is poorly calibrated and MLLS-style estimation is used without correcting for that, when target samples are too small to produce stable estimates, or when the observed change actually stems from a data-pipeline error rather than a real distributional shift.
How often should label shift be monitored?
There is no universal cadence; it depends on how quickly the business or clinical context can plausibly change and how costly a missed shift would be. Many teams pair continuous automated monitoring of predicted-class frequencies with periodic, scheduled deeper reviews (using any labeled data that has accumulated), and add ad hoc reviews around known seasonal events or major operational changes.
Can label shift occur in regression?
Classical label shift, as formalized by [Lipton et al. (2018)][1] and [Saerens et al. (2002)][2], is framed specifically for classification with discrete class labels and relies on a class-conditional stability assumption that doesn't directly translate to a continuous target variable. Related ideas — changes in the marginal distribution of a continuous target between training and deployment — are studied in regression and forecasting under different names and different assumption structures, and shouldn't be assumed to inherit the same guarantees or correction formulas discussed here.
What happens when a new class appears?
If a class exists in the target domain with positive probability but was entirely absent from the source training data, no amount of prior reweighting can recover it, since the model never learned any representation of what that class looks like. This is a genuine identifiability limit, not a solvable estimation problem, and typically requires new labeled data and retraining rather than a correction formula.
How can teams validate that a correction worked?
The most reliable approach uses labeled target data, as it becomes available (even delayed or partial), to directly compare log loss, calibration error, and cost-sensitive metrics between the uncorrected model, the corrected model, a recalibrated model, and a simple baseline. Running the correction in shadow mode before full deployment, and cross-checking prior estimates from more than one method (such as BBSE and MLLS together), adds further confidence before relying on the correction for real decisions.
Key Takeaways
Label shift means class priors P(Y) change between training and deployment while class-conditional feature distributions P(X∣Y) stay stable — a specific, mathematically tractable slice of the broader dataset-shift problem.
The posterior-adjustment formula reweights a model's existing outputs by the ratio of new to old class priors, then renormalizes — no retraining required, if the assumption holds.
BBSE, MLLS/EM, and RLLS are the leading estimation methods for recovering unknown target priors from unlabeled data, each with different assumptions and failure modes.
A change in observed class frequencies is evidence of possible label shift, not proof of it; covariate shift, concept shift, and pipeline errors can all produce similar surface signals.
Calibration quality materially affects estimation accuracy, particularly for likelihood-based methods like MLLS.
New target classes, near-singular confusion matrices, and very small samples are hard identifiability limits, not just estimation noise to be regularized away.
Correction should be validated in shadow mode against labeled data before being trusted for live decisions, since a mathematically valid correction under a wrong assumption can worsen performance.
Ongoing production monitoring — not a single detection-and-correction event — is what keeps label-shift handling reliable over time.
Actionable Next Steps
Pull your model's held-out source-domain confusion matrix and current class-prior estimates as a documented baseline.
Set up automated monitoring of predicted-class frequencies against that baseline, including seasonally-aware alert thresholds.
When a shift is flagged, run a BBSD-style or two-sample statistical test before assuming label shift specifically.
Cross-check target prior estimates using at least two methods (for example, BBSE and MLLS) and inspect their confidence intervals.
Apply the posterior-adjustment formula in shadow mode, logging both corrected and uncorrected outputs alongside model version and timestamp.
As delayed or spot-checked labels arrive, compare log loss, calibration error, and cost-sensitive metrics across the uncorrected, corrected, recalibrated, and baseline models.
Document the decision, update the reference window, and schedule the next scheduled review rather than treating the fix as permanent.
Glossary
Bayes' theorem: A rule relating a class's posterior probability to its prior probability and the likelihood of the observed features given that class.
Black Box Shift Detection (BBSD): A method for testing whether label shift has occurred by checking whether a classifier's predicted-label distribution has changed, exploiting an invertible confusion matrix.
Black Box Shift Estimation (BBSE): A method for estimating new target class priors using a classifier's confusion matrix and its predictions on unlabeled target data.
Calibration: The property that a model's predicted probabilities match observed outcome frequencies.
Class-conditional distribution: The distribution of features given a specific class label, written P(X∣Y).
Class imbalance: An unequal proportion of classes within a single dataset, distinct from label shift, which describes a change between datasets.
Class prior: The overall probability of a class occurring, written P(Y).
Concept shift (conditional shift): A change in the relationship between features and labels itself, P(Y∣X).
Confusion matrix: A table summarizing how often a classifier predicts each label given each true label.
Covariate shift: A change in the feature distribution P(X) while the posterior relationship P(Y∣X) is assumed stable.
Dataset shift: The broad umbrella term for any change in the joint distribution of features and labels between training and deployment.
Expectation-maximization (EM): An iterative algorithm that alternates between estimating hidden parameters and refining estimates based on them until convergence; used in MLLS-style label-shift estimation.
Identifiability: Whether a quantity can, in principle, be uniquely recovered from available data; a target class absent from source data is not identifiable through prior correction alone.
Importance weight, w(y): The ratio of a class's target-domain prior to its source-domain prior, used to reweight posteriors.
Label noise: Incorrect labels in a dataset, distinct from label shift.
Label shift: A change in class priors P(Y) between source and target domains while class-conditionals P(X∣Y) remain stable; also called class-prior shift or prior probability shift.
Maximum-likelihood label-shift estimation (MLLS): A method estimating target class priors by maximizing the likelihood of observed unlabeled target features under the model.
Model monitoring: The ongoing practice of tracking a deployed model's inputs, outputs, and performance over time.
Posterior probability: The probability of a class given observed features, P(Y∣X).
Regularized Learning under Label Shifts (RLLS): An importance-weighting method with an explicit generalization bound, designed to be robust in small-sample regimes.
Sampling bias: Non-representative data collection that distorts the observed distribution relative to the true population.
Source domain: The distribution from which training and validation data were drawn.
Target domain: The distribution a model actually encounters after deployment.
Sources & References
[1] Lipton, Zachary C., Yu-Xiang Wang, and Alexander J. Smola. "Detecting and Correcting for Label Shift with Black Box Predictors." Proceedings of the 35th International Conference on Machine Learning, PMLR, vol. 80, 2018, pp. 3122–3130. https://proceedings.mlr.press/v80/lipton18a.html
[2] Saerens, Marco, Patrice Latinne, and Christine Decaestecker. "Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure." Neural Computation, vol. 14, no. 1, 2002, pp. 21–41. DOI: 10.1162/089976602753284446. https://direct.mit.edu/neco/article/14/1/21/6577/Adjusting-the-Outputs-of-a-Classifier-to-New-a
[3] Garg, Saurabh, Yifan Wu, Sivaraman Balakrishnan, and Zachary C. Lipton. "A Unified View of Label Shift Estimation." Advances in Neural Information Processing Systems 33 (NeurIPS 2020). https://proceedings.neurips.cc/paper/2020/hash/219e052492f4008818b8adb6366c7ed6-Abstract.html
[4] Azizzadenesheli, Kamyar, Anqi Liu, Fanny Yang, and Animashree Anandkumar. "Regularized Learning for Domain Adaptation under Label Shifts." International Conference on Learning Representations (ICLR), 2019. https://arxiv.org/abs/1903.09734
[5] Rabanser, Stephan, Stephan Günnemann, and Zachary C. Lipton. "Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift." Advances in Neural Information Processing Systems 32 (NeurIPS 2019), pp. 1394–1406. https://arxiv.org/abs/1810.11953
[6] Quiñonero-Candela, Joaquin, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence, editors. Dataset Shift in Machine Learning. MIT Press, 2008. https://direct.mit.edu/books/edited-volume/3841/Dataset-Shift-in-Machine-Learning
[7] Alexandari, Amr, Anshul Kundaje, and Avanti Shrikumar. "Maximum Likelihood with Bias-Corrected Calibration Is Hard-to-Beat at Label Shift Adaptation." Proceedings of the 37th International Conference on Machine Learning, PMLR, vol. 119, 2020, pp. 222–232. https://proceedings.mlr.press/v119/alexandari20a.html
[8] Google Search Central. "Article (Article, NewsArticle, BlogPosting) Structured Data." Google Developers. https://developers.google.com/search/docs/appearance/structured-data/article
[9] Schema.org. "FAQPage." https://schema.org/FAQPage
