492 lines
28 KiB
TeX
492 lines
28 KiB
TeX
% ============================================================
|
||
% PLACEMENT: near the START of Ch. 2, before KDE and AUC/lit
|
||
% review sections. This is the reader's first encounter with
|
||
% the problem setting and the TPR@FPR objective. The arc
|
||
% should be: (1) standard accuracy under imbalance fails;
|
||
% (2) class-weighting is the usual fix but gives no direct
|
||
% operating-point control; (3) define TPR, FPR, ROC formally;
|
||
% (4) state the target: maximise TPR at a fixed FPR budget α.
|
||
% ============================================================
|
||
\section{Imbalanced Classification and The TPR/FPR Objective}
|
||
\label{sec:imbalanced-tpr-fpr}
|
||
|
||
% IMPROVEMENT SUGGESTIONS (carried over from original draft):
|
||
% - Narrative arc is currently backwards. Drive the section
|
||
% with: accuracy is gameable under imbalance -> class
|
||
% weighting is the standard fix -> it gives no direct
|
||
% control over the operating point -> therefore we need
|
||
% TPR@FPR directly.
|
||
% - The inseparability tangent (cascade-classifier example)
|
||
% dilutes the argument; trim to one sentence or cut.
|
||
% - $\mathcal{D}_0$ and $\mathcal{D}_1$ must be defined
|
||
% before first use.
|
||
% - Land the section on a concrete, formal statement of the
|
||
% TPR@FPR objective with proper notation.
|
||
|
||
When fitting a classifier, it is most common to try and maximize the probability
|
||
of success. That is, given some dataset
|
||
$\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^{n}$
|
||
with inputs $\mathbf{x}_i \in \mathcal{X}$ and binary labels $y_i \in \{0, 1\}$, we learn a scoring function $f_\theta : \mathcal{X} \to \mathbb{R}$ and apply a threshold $\tau$ to produce predictions $\hat{y} = \mathbf{1}[f_\theta(\mathbf{x}) > \tau]$. The standard objective is to minimize the probability of classification error:
|
||
|
||
$$\mathcal{L}(\theta) = \frac{1}{n} \sum_{i=1}^{n} \mathbf{1}[\hat{y}_i \neq y_i] = P(\hat{y} \neq y)$$
|
||
|
||
which is equivalent to maximizing classification accuracy $P(\hat{y} = y)$.
|
||
|
||
For most applications, this optimization is warranted. However, there are many real-world settings in which
|
||
this type of fitting is ill-suited.
|
||
|
||
One such application is in the case of inseparable classes. In many cases, no meaningful
|
||
separation boundary can be found between $\mathcal{D}_0$ and $\mathcal{D}_1$. In many cases,
|
||
especially in tabular datasets, there can be two data elements $i,j$ for which $\mathbf{x}_i = \mathbf{x}_j$,
|
||
however, $y_i \neq y_j$. Any deterministic classifier will have to decide whether to classify
|
||
inseparable points as one class or the other, but it remains hard to control the behavior of
|
||
a classifier while it is fitting to suggest one outcome or the other.
|
||
|
||
The issue of inseparability becomes more apparent in certain applications where runtime is a key
|
||
and limiting factor. In many classification pipelines, a ``quick and dirty'' classifier may
|
||
filter out many candidates before an ``in depth'' classifier gives the final answer. This may be done
|
||
in events where running the full in-depth classification on all data points is not a cost-effective or time-efficient solution.
|
||
The ``quick and dirty'' classifier can be either low in parameters, making it run quickly. It may also only
|
||
make a quick decision based on part of the data point. For example, in the case of a classifier looking to flag
|
||
images of the ocean, a ``quick and dirty'' classifier may look at a very-low resolution version of the image and simply check if it is relatively ``blue''. It filters out many certain false candidates this way, then passes on the remainder to an in-depth, higher accuracy classifier. Solutions like this allow to run at near-optimal performance with lower compute resources in limited-time applications. These situations are common in large-volume image applications on the internet, as well as in astronomy/microscopy.
|
||
|
||
Another such application is a case of highly imbalanced data. Even with separability, it is very
|
||
easy to a classifier to simply ``ignore'' the minority class. This may very likely be a local minimum
|
||
while fitting, which may be inescapable. As was once said, ``a classifier that always returns 0 is
|
||
well fitted for flagging pictures of Michael Jordan bathing in green M\&Ms''. Combined with inseparability, we see that in extreme cases, classifiers may choose to ignore, or otherwise penalize, the minority class.
|
||
|
||
For every trained classifier, the output per model is a score (TODO rewrite and formalize).
|
||
|
||
TODO finish this section
|
||
|
||
\begin{itemize}
|
||
\item What usually happens
|
||
\begin{itemize}
|
||
\item Fitting
|
||
\item Graphing ROC and picking threshold based on tradeoff
|
||
\end{itemize}
|
||
\item Usually we compensate for poor performance with
|
||
\begin{itemize}
|
||
\item Weighting between classes
|
||
\item Possible other methods
|
||
\end{itemize}
|
||
\item Problem with this method is lack of control.
|
||
\end{itemize}
|
||
|
||
|
||
% Motivate why standard loss functions (cross-entropy, MSE) break down under class
|
||
% imbalance: they optimize average accuracy, which a model can game by predicting
|
||
% the majority class. Introduce concrete examples (e.g. medical diagnosis, fraud
|
||
% detection) where the cost of a false negative vastly outweighs a false positive,
|
||
% and where the practitioner needs to operate at a specific FPR budget. Argue that
|
||
% what is actually needed is direct control over TPR at a fixed FPR, not a
|
||
% surrogate that only loosely correlates with it.
|
||
|
||
|
||
% ============================================================
|
||
% PLACEMENT: Section 2.2, immediately after the imbalanced
|
||
% classification section (sec:imbalanced-tpr-fpr). Establishes
|
||
% AUC as the standard measure and surveys how the literature
|
||
% optimizes it, before narrowing to partial AUC / single-point
|
||
% objectives in the next section.
|
||
% ============================================================
|
||
\section{ROC Curves and AUC Optimization}
|
||
\label{sec:roc-auc}
|
||
\addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{עקומות ROC ואופטימיזציה של AUC}}
|
||
|
||
% \subsection{The ROC Curve}
|
||
% Define the ROC curve as the plot of TPR vs.\ FPR as the threshold sweeps its full range.
|
||
% Show that AUC equals $P(f(x^+) > f(x^-))$ for a random positive/negative pair (the
|
||
% Wilcoxon--Mann--Whitney statistic): AUC is threshold-independent and measures ranking
|
||
% quality, not calibration. Cite Hanley \& McNeil (1982).
|
||
|
||
% \subsection{Surrogate AUC Losses}
|
||
% Explain why directly maximizing the empirical AUC is hard: the WMW indicator is pairwise
|
||
% and discontinuous ($O(n^2)$ pairs). Introduce smooth surrogates --- sigmoid approximation
|
||
% to the pairwise indicator, squared loss on score differences --- and their consistency
|
||
% guarantees. Cite Cortes \& Mohri (2003/2004).
|
||
|
||
% \subsection{Stochastic AUC Maximization}
|
||
% Describe mini-batch and compositional stochastic methods that make AUC optimization
|
||
% practical at scale. Cover PESG (Yuan et al., NeurIPS 2020) and LibAUC (Yuan et al.,
|
||
% ICML 2021), which is used as a baseline in \Cref{chapter:experiments}.
|
||
% Key limitation shared by all: they optimize the \emph{whole-curve} AUC, not a specific
|
||
% operating point on the ROC --- motivating the next section.
|
||
|
||
|
||
\subsection{AUC and ROC definitions}
|
||
|
||
% TODO add formalizations
|
||
% Formalize ROC and integration into AUC
|
||
|
||
A Receiver Operating Characteristic (ROC) is a graph that represents the performance of a classifier.
|
||
The classifier in question is evaluated on binary-labeled data. The labeled data is ordered based off of the
|
||
scores given by the classifier. Given any numerical threshold, it is then easy to see which labeled data points fall above and below.
|
||
|
||
For each threshold, the true positive rate and the false positive rate is given. The results for each threshold, and so for each possible TPR and FPR, are put on a graph. The resulting curve is the ROC curve.
|
||
|
||
The AUC is a number in the range $[0, 1]$ representing the total area under the curve. It is calculated by integrating over all TPRs and taking the resulting average FPRs. A result of $1$ represents a perfect classifier,
|
||
while a result of $0$ represents a classifier with a perfect opposite result. A result of $0.5$ represents a classifier that found no separation whatsoever.
|
||
|
||
\subsection{AUC as a metric of model performance}
|
||
|
||
The AUC is a direct representation of the performance of a classifier, as opposed to a loss. There are instances where the loss may be minimized, yet seperation might be minimal. These two metrics might correlate, but do not necessarily correspond.
|
||
|
||
An example of this is a result for which all true samples receive a score of $0.51$ and all false samples receie a score of $0.49$. Using many losses, including BCE, this would result in a relatively high loss value. On this score however, the AUC is $1$. The model perfectly seperates the samples, so further training is not necessary.
|
||
|
||
Direct AUC optimization can be ideal with optimizing for imbalanced datasets. With most losses such as BCE or focal, the loss is calculated per-sample and then summed (or averaged). In order to properly train for both true and false populations, the scores are weighted, so that equal weight falls on both the true and the false populations. With direct AUC optimization, the imbalance does not affect the calculation, and the reweighting can be forgone.
|
||
|
||
Additionally, direct AUC methods are more resilient to label noise, as opposed to classical methods. Due to inseperability, correctly classifying a sample might require a model to incorrectly classify similar or even identical samples that were mislabeled. This is only worth it for the model in training if the total loss of the correctly labeled samples outweighs the total loss of the incorrectly labeled samples. For most methods, the weight per class is given by the class imbalance in the dataset, but this doesn't take into account the label noise in any way. Optimizing for AUC specifically would eliminate this risk.
|
||
|
||
\subsection{AUC optimization in literature}
|
||
|
||
While AUC as a metric for model performance has existed for a while, methods to optimize for it directly are relatively recent.
|
||
%TODO add references
|
||
|
||
% ============================================================
|
||
% PLACEMENT: Section 2.3, after ROC/AUC section above and
|
||
% before the KDE background section. Narrows from whole-curve
|
||
% AUC to the single operating-point objective that ROLL targets.
|
||
% ============================================================
|
||
\section{Partial AUC and TPR at Fixed FPR}
|
||
\label{sec:partial-auc}
|
||
\addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{AUC חלקי ו-TPR בנקודת FPR קבועה}}
|
||
|
||
% \subsection{Partial AUC}
|
||
% Define partial AUC as the area under the ROC curve restricted to $\mathrm{FPR} \in [0, \alpha]$.
|
||
% Motivate: in high-stakes settings (fraud, medical triage) only the low-FPR region matters.
|
||
% Survey recent pAUC optimizers that extend the PESG/compositional framework
|
||
% (Lin et al., 2022; Yang \& Ying, 2022).
|
||
|
||
% \subsection{TPR at a Fixed FPR: The Single-Point Objective}
|
||
% Distinguish pAUC (integral over an interval) from the exact single-point objective:
|
||
% maximize $\mathrm{TPR}(f, \alpha)$ for a fixed budget $\alpha$.
|
||
% Argue that post-hoc threshold search and pAUC integration both fail to provide a
|
||
% differentiable end-to-end training signal for this precise target --- this is the gap
|
||
% that ROLL fills.
|
||
|
||
% \subsection{Neyman-Pearson Classification (Brief)}
|
||
% Introduce NP classification (Cannon et al., 2002; Tong, 2013) as the statistical
|
||
% framing: minimize type-II error subject to $\mathrm{FPR} \le \alpha$.
|
||
% Note that classical NP methods are not gradient-based and cannot be applied to DNNs.
|
||
% Full treatment deferred to \Cref{sec:neyman-pearson} at the end of this chapter.
|
||
|
||
TODO write this section.
|
||
|
||
|
||
% ============================================================
|
||
% PLACEMENT: Section 2.4, after partial AUC and before the
|
||
% noisy labels section. Provides the technical KDE background
|
||
% that Ch.\ 3 (sec:roll-kde) builds on directly. End with a
|
||
% forward pointer to sec:roll-kde.
|
||
% ============================================================
|
||
\section{Kernel Density Estimation}
|
||
\label{sec:kde-background}
|
||
\addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{אמידת צפיפות גרעין (KDE)}}
|
||
|
||
% \subsection{The Parzen Window Estimator}
|
||
% Define KDE: $\hat{p}(x) = \frac{1}{nh}\sum_i K\!\bigl(\frac{x-x_i}{h}\bigr)$,
|
||
% where $K$ is a kernel and $h > 0$ is the bandwidth.
|
||
% Present common kernels: Gaussian, Epanechnikov, sigmoid. Discuss how kernel
|
||
% smoothness affects gradient properties. Cite Parzen (1962), Silverman (1986).
|
||
|
||
% \subsection{Bandwidth Selection}
|
||
% Describe Silverman's rule-of-thumb ($h = 1.06\,\hat\sigma\,n^{-1/5}$) as the default.
|
||
% Mention cross-validation as the principled alternative, expensive at training time.
|
||
% In ROLL, bandwidth is fixed (Silverman) or treated as a learnable parameter;
|
||
% details deferred to Ch.\ 3.
|
||
|
||
% \subsection{Differentiability and Use as a Loss Ingredient}
|
||
% Explain why KDE with a smooth kernel is differentiable w.r.t.\ the support points $x_i$:
|
||
% the density at any query is a smooth weighted sum over all support points, so gradients
|
||
% flow back to network parameters. This property is why KDE is suitable as a building
|
||
% block inside a loss function. Forward pointer: \Cref{sec:roll-kde} exploits this to
|
||
% construct ROLL's differentiable threshold estimate from the estimated score distributions.
|
||
|
||
When describing the source distribution for samples, rarely do they conform to some parametric distribution.
|
||
Some data, such as adult heights, follows a Gaussian-like distribution; the occurrence of radioactive decay
|
||
events or car arrivals at an intersection follows a Poisson distribution. For arbitrary real-world data,
|
||
however, no such assumption can be made safely.
|
||
|
||
This motivates Kernel Density Estimation (KDE). Rather than assuming a parametric form, KDE uses the
|
||
empirical sample itself as the model: each observed point $x_i$ contributes a smooth bump --- a
|
||
\emph{kernel} --- to the estimated density. The assumption is that nearby points are plausibly likely,
|
||
so probability mass is spread locally around each sample. Summing all kernels and normalizing yields a
|
||
smooth PDF that mirrors the shape of the underlying distribution without committing to a parametric family.
|
||
|
||
Formally, let $\{x_i\}_{i=1}^n$ be the observed samples, $K$ a kernel function satisfying
|
||
$\int_{-\infty}^{\infty} K(u)\,du = 1$, and $h > 0$ the \emph{bandwidth} controlling the spread of
|
||
each kernel. Setting the correct bandwidth is important - too low a bandwidth parameter and the KDE becomes too jagged, and the probability anywhere there was no candidate approaches zero, avoiding generalization. Too high a bandwidth parameter, on the other hand, smooths out the distribution entirely, and the entire distribution starts to approach that of the kernel itself, losing the specificity that the candidates bring.
|
||
|
||
The KDE density estimator is:
|
||
\[
|
||
\hat{p}(x) = \frac{1}{nh} \sum_{i=1}^{n} K\!\left(\frac{x - x_i}{h}\right)
|
||
\]
|
||
Each term $\frac{1}{h}K\!\bigl(\frac{x-x_i}{h}\bigr)$ is a kernel centered at $x_i$; the
|
||
$1/h$ factor compensates for the change of variables and ensures the full sum integrates to 1.
|
||
|
||
Valid kernels must be non-negative, symmetric around zero, and normalized. These differ in smoothness, tail behavior, and computational cost. A particularly useful choice is
|
||
the \emph{logistic kernel}:
|
||
\[
|
||
K(u) = \frac{e^{-u}}{(1 + e^{-u})^2}
|
||
\]
|
||
which is smooth, bell-shaped, and centered at zero. Its key property is that its CDF has a closed form,
|
||
which is the sigmoid function:
|
||
\[
|
||
\int_{-\infty}^{x} K(u)\,du \;=\; \frac{1}{1+e^{-x}} \;=:\; \sigma(x)
|
||
\]
|
||
This gives the KDE cumulative distribution function a closed form as well:
|
||
\[
|
||
\hat{F}(\tau) = \frac{1}{n}\sum_{i=1}^{n} \sigma\!\left(\frac{\tau - x_i}{h}\right)
|
||
\]
|
||
|
||
|
||
% ============================================================
|
||
% PLACEMENT: Section 2.5, AFTER the KDE background section
|
||
% (sec:kde-background) and BEFORE the Neyman-Pearson bridge
|
||
% (sec:neyman-pearson, which becomes 2.6).
|
||
%
|
||
% Purpose: establish that existing pointwise losses are
|
||
% fragile under label corruption, motivating both the
|
||
% practical robustness of ROLL and the poisoning experiment
|
||
% results in Ch. 4.
|
||
%
|
||
% Narrative arc:
|
||
% 1. Define label noise types: symmetric (uniform flip),
|
||
% asymmetric / class-conditional (only certain class
|
||
% transitions occur), instance-dependent. Our poisoning
|
||
% experiment is asymmetric: P(\tilde{y}=0 \mid y=1)>0,
|
||
% P(\tilde{y}=1 \mid y=0)=0.
|
||
% 2. Why BCE is fragile: the symmetry condition (Ghosh
|
||
% et al., AAAI 2017) — a loss \ell is noise-tolerant
|
||
% iff \sum_k \ell(f(x),k) = C (constant). CE fails
|
||
% this; MAE satisfies it but converges slowly. Cite
|
||
% Feng et al. (IJCAI 2020) for "Can CE be robust?"
|
||
% (only under very strict conditions). Implication:
|
||
% mislabeled samples produce large unmitigated
|
||
% gradients; DNNs memorize noise late in training
|
||
% (memorization effect).
|
||
% 3. Family of existing fixes — keep brief; these are what
|
||
% ROLL sidesteps structurally:
|
||
% (a) Robust loss functions: MAE (noise-tolerant,
|
||
% slow), GCE (Zhang & Sabuncu, NeurIPS 2018;
|
||
% interpolates MAE↔CE via q), asymmetric losses
|
||
% (Zhou et al., ICML 2021), symmetrization
|
||
% (arXiv:2605.20347).
|
||
% (b) Label correction via noise transition matrix
|
||
% (Patrini et al., CVPR 2017): estimate
|
||
% T[i,j]=P(\tilde{y}=j|y=i), correct logits by
|
||
% T^{-1}. Hard for asymmetric noise without a
|
||
% clean reference set.
|
||
% (c) Sample selection / co-teaching (Han et al.,
|
||
% NeurIPS 2018): two networks select small-loss
|
||
% examples for each other, exploiting the
|
||
% memorization effect. Requires dual-model
|
||
% training.
|
||
% 4. Ranking / distributional objectives and noise
|
||
% robustness: pairwise losses depend on score
|
||
% differences, not absolute labels — one mislabeled
|
||
% sample affects O(n) pairs, each with a small gradient
|
||
% weight. Wang et al. (arXiv:2305.14258) formally show
|
||
% noise robustness of partial-AUC objectives.
|
||
% 5. Bridge (observation, not oversell): ROLL's KDE-based
|
||
% distributional objective shares this structural
|
||
% property. Fake negatives (copies of positives with
|
||
% flipped labels) shift the estimated negative density
|
||
% but do not erase distributional separation when true
|
||
% positive scores remain clustered. ROLL achieves this
|
||
% without any explicit noise-handling mechanism; the
|
||
% poisoning experiment in Ch. 4 provides empirical
|
||
% evidence.
|
||
% ============================================================
|
||
\section{Learning with Noisy Labels}
|
||
\label{sec:noisy-labels}
|
||
|
||
% SUBSECTION STRUCTURE:
|
||
%
|
||
% \subsection{Types of Label Noise}
|
||
% - Symmetric / uniform: each label flipped with
|
||
% probability \varepsilon across all classes.
|
||
% - Asymmetric / class-conditional: only certain class
|
||
% transitions occur. Our poisoning experiment is of
|
||
% this type: P(\tilde{y}=0 \mid y=1)>0,
|
||
% P(\tilde{y}=1 \mid y=0)=0.
|
||
% - Instance-dependent: flip probability varies per
|
||
% sample (hardest to handle; mention only briefly).
|
||
% - Note: even moderate noise rates significantly hurt
|
||
% pointwise losses in the imbalanced setting, because
|
||
% mislabeled positives are rare to begin with.
|
||
%
|
||
% \subsection{Why Cross-Entropy Is Fragile Under Label Noise}
|
||
% - Introduce symmetry condition \cite{ghosh2017robust}:
|
||
% a loss is noise-tolerant iff
|
||
% \sum_k \ell(f(\mathbf{x}), k) = C (constant).
|
||
% - CE does NOT satisfy this; MAE does (but is slow).
|
||
% - Memorization effect: DNNs learn clean patterns early,
|
||
% then memorize noisy labels late in training.
|
||
% - Cite: \cite{ghosh2017robust}, \cite{feng2020can}.
|
||
%
|
||
% \subsection{Approaches to Robust Learning}
|
||
% - Paragraph 1 — Robust loss functions:
|
||
% MAE \cite{ghosh2017robust},
|
||
% GCE \cite{zhang2018generalized} (L_q interpolation),
|
||
% asymmetric losses \cite{zhou2021asymmetric}.
|
||
% Key tradeoff: noise-tolerance vs.\ convergence speed.
|
||
% - Paragraph 2 — Transition matrix correction
|
||
% \cite{patrini2017making}: forward correction
|
||
% multiplies logits by T^{-1}. Canonical method but
|
||
% requires estimating T, which is hard for asymmetric
|
||
% noise without a clean anchor set.
|
||
% - Paragraph 3 — Sample selection / co-teaching
|
||
% \cite{han2018coteaching}: two nets select small-loss
|
||
% samples for each other each epoch. Bypasses T
|
||
% estimation but requires dual-model training.
|
||
%
|
||
% \subsection{Ranking Objectives and Noise Robustness}
|
||
% - Pairwise ranking losses are structurally more
|
||
% noise-robust: a single mislabeled sample affects O(n)
|
||
% pairs, each with small gradient weight, vs.\ one large
|
||
% pointwise gradient in BCE.
|
||
% - Cite Wang et al.\ \cite{wang2023weakly} for formal
|
||
% treatment in the partial-AUC setting.
|
||
% - Close with bridge sentence: ROLL's distributional
|
||
% threshold objective operates in the same structural
|
||
% regime — not pairwise, but also not pointwise —
|
||
% conferring noise robustness without explicit correction.
|
||
% Empirical evidence deferred to the Experiments chapter.
|
||
|
||
While collecting data with correct labels should be a priority, in some cases it is impossible.
|
||
Many data collection methods lead to imperfect labeling, resulting in a muddy ground truth.
|
||
This could be due to a number of factors. The labels might be based on human labeling, which is error prone.
|
||
The labeling might be true, yet the signal might be noisy or wrong, creating a mismatch between signal and label.
|
||
The labeling could be a resource-intensive process, and so it might only be done for a portion of the data.
|
||
And in some cases, the labeling is done based off a different learned model, which too could be faulty.
|
||
|
||
\subsection{Label noise formalization}
|
||
|
||
Given a dataset $ \mathcal{D} = \{(x_i, y_i)\} $, we would define the noisy version as $ \tilde{\mathcal{D}} = \{(x_i, \tilde{y_i})\} $. The noise $N_i$ is defined as:
|
||
|
||
\[
|
||
N_i =
|
||
\begin{cases}
|
||
0 & \text{if } y_i = \tilde{y_i} \\
|
||
1 & \text{otherwise}
|
||
\end{cases}
|
||
\]
|
||
|
||
Label noise is usually categorized as being symmetric, asymmetric or instance dependent.
|
||
|
||
For instance dependent noise, $\mathbb{P}(N_i = 1)$ depends on $x_i$. That is, the probability that a label is flipped depends on $x_i$. For the other two types, the probability of the label being flipped depends only on the original label itself.
|
||
|
||
For symmetric noise, $\mathbb{P}(N_i = 1 | y_i = 1) = \mathbb{P}(N_i = 1 | y_i = 0)$. The probability of the label being flipped does not depend on the original label. This is unlike asymmetric noise, for which the probability of the label flipping does depend on the original label.
|
||
|
||
\subsection{Existing approaches}
|
||
|
||
Prior work has been done in order to make noise-robust learning processes. These can be categorized into three main approaches.
|
||
|
||
\subsubsection{Label meta-training and adjustment}
|
||
Under this approach, the training process is adjusted by modelling label noise explicitly.
|
||
\citet{patrini2017making} introduced forward and backward loss correction: by estimating a
|
||
noise transition matrix $T$, where $T_{ij} = P(\tilde{y}=j \mid y=i)$, the observed noisy
|
||
loss can be corrected to recover the clean-label risk.
|
||
Estimating $T$ requires identifying high-confidence \emph{anchor points} for each class from
|
||
a pre-trained noisy model --- a two-phase procedure that may fail under high or asymmetric
|
||
noise rates.
|
||
\citet{ren2018learning} sidestep transition matrix estimation by using a small clean
|
||
validation set and meta-gradient descent to assign per-sample loss weights at each training
|
||
step, adapting dynamically to whatever noise is present without assuming a fixed noise
|
||
structure.
|
||
Both approaches require either prior knowledge of the noise model or access to a clean
|
||
held-out set, which is not always available in practice.
|
||
|
||
\subsubsection{Sample selection}
|
||
Rather than modifying the loss, sample selection methods filter likely-noisy examples during
|
||
training.
|
||
The approach is grounded in the \emph{memorization effect} documented by
|
||
\citet{arpit2017memorization}: deep networks learn clean-label patterns first and memorize
|
||
noisy labels only later in training.
|
||
This means that at early training stages, samples with small loss are more likely to be
|
||
correctly labelled and can be preferentially retained.
|
||
\citet{han2018coteaching} exploit this with Co-teaching: two networks train in parallel,
|
||
each selecting its own small-loss samples and passing them to the other network for the
|
||
parameter update, preventing both from converging to the same memorized errors.
|
||
\citet{li2020dividemix} extend the idea by fitting a per-class Gaussian mixture model to
|
||
the loss distribution at each epoch to separate clean from noisy samples, then treating the
|
||
noisy partition as unlabeled data within a semi-supervised learning framework.
|
||
These methods can achieve strong performance but substantially increase training complexity,
|
||
requiring multiple models or multi-phase procedures.
|
||
|
||
\subsubsection{Robust loss functions}
|
||
A third line of work designs loss functions that are intrinsically resistant to label noise,
|
||
requiring no changes to the training procedure beyond swapping the loss term.
|
||
\citet{ghosh2017robust} established a sufficient condition for noise tolerance: a loss
|
||
$\ell$ tolerates uniform symmetric noise if and only if
|
||
|
||
\[
|
||
\sum_{k=1}^{K} \ell(f(\mathbf{x}), k) = C \quad \text{for all } \mathbf{x},\, f
|
||
\]
|
||
|
||
for some constant $C$ independent of $f(\mathbf{x})$.
|
||
Intuitively, if the total loss over all class assignments is constant, randomly flipping
|
||
labels cannot shift the expected risk minimizer.
|
||
Mean Absolute Error (MAE) satisfies this condition --- one can verify that
|
||
$\sum_k (1 - f_k(\mathbf{x})) = K - 1$ for a $K$-class softmax output ---
|
||
whereas cross-entropy does not.
|
||
However, MAE converges slowly in practice because its gradient has constant magnitude
|
||
regardless of prediction confidence, making it insensitive to easy examples that should
|
||
receive small updates.
|
||
|
||
\citet{zhang2018generalized} proposed the Generalised Cross Entropy (GCE) loss, defined via
|
||
the $L_q$ family:
|
||
|
||
\[
|
||
\ell_q(f(\mathbf{x}), y) = \frac{1 - f_y(\mathbf{x})^q}{q}, \quad q \in (0,\,1]
|
||
\]
|
||
|
||
where $f_y(\mathbf{x})$ is the model's predicted probability for the true class $y$.
|
||
As $q \to 0$ the loss recovers cross-entropy (via L'H\^{o}pital's rule); at $q = 1$ it
|
||
reduces to MAE, which satisfies the symmetry condition.
|
||
Intermediate values therefore trade noise tolerance for faster convergence.
|
||
We evaluate GCE at $q = 0.7$ as a baseline in the Experiments chapter.
|
||
\citet{zhou2021asymmetric} extend the theoretical framework to asymmetric noise, introducing
|
||
the \emph{asymmetry ratio} to quantify how noise tolerance degrades when flip rates differ
|
||
between classes.
|
||
\citet{feng2020can} confirm that standard cross-entropy cannot be made noise-tolerant
|
||
through simple modification, establishing robust losses as a genuinely distinct family
|
||
rather than a patch on existing objectives.
|
||
|
||
All three families above treat noise as a quantity to be explicitly corrected --- by
|
||
adjusting labels, filtering samples, or redesigning the loss.
|
||
ROLL takes a different route: its distributional threshold objective aggregates over the
|
||
full score distribution of each class, so a small fraction of mislabelled samples shifts
|
||
the estimated density slightly but cannot dominate the threshold.
|
||
This structural robustness arises from the distributional formulation itself, without any
|
||
explicit noise-handling mechanism; empirical evidence is provided in the Experiments
|
||
chapter.
|
||
|
||
% ============================================================
|
||
% PLACEMENT: Section 2.6 (was 2.5), near the END of Ch. 2,
|
||
% after KDE and Noisy Labels sections, just before the
|
||
% chapter summary. This section bridges from prior work to
|
||
% the method chapter by framing ROLL as a learnable,
|
||
% gradient-based NP test.
|
||
% NOTE: the label sec:neyman-pearson is unchanged; existing
|
||
% \Cref references in Ch. 3 (sec:problem-formulation) still
|
||
% work without modification.
|
||
% ============================================================
|
||
\section{Connection to Neyman-Pearson}
|
||
\label{sec:neyman-pearson}
|
||
|
||
% Introduce the Neyman-Pearson lemma: among all tests at a
|
||
% given false-positive rate, the likelihood-ratio test
|
||
% maximizes TPR. Frame ROLL as a learnable, gradient-based
|
||
% realization of this principle — rather than assuming known
|
||
% class-conditional distributions (as classical NP does),
|
||
% ROLL estimates them from model scores during training.
|
||
% Briefly note that existing NP-inspired methods do not
|
||
% support deep learning / gradient-based optimization; this
|
||
% is the gap ROLL fills.
|
||
|
||
TODO write this section.
|
||
|