335 lines
21 KiB
TeX
335 lines
21 KiB
TeX
% ============================================================
|
|
% PLACEMENT: Section 2.1 — opens Chapter 2. Establishes AUC
|
|
% as the standard measure for ranking-based classification and
|
|
% surveys how the literature optimizes it directly. Sets up
|
|
% the noise robustness discussion in the next section.
|
|
% ============================================================
|
|
\section{ROC Curves and AUC Optimization}
|
|
\label{sec:roc-auc}
|
|
\addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{עקומות ROC ואופטימיזציה של AUC}}
|
|
|
|
% \subsection{The ROC Curve}
|
|
% Define the ROC curve as the plot of TPR vs.\ FPR as the threshold sweeps its full range.
|
|
% Show that AUC equals $P(f(x^+) > f(x^-))$ for a random positive/negative pair (the
|
|
% Wilcoxon--Mann--Whitney statistic): AUC is threshold-independent and measures ranking
|
|
% quality, not calibration. Cite Hanley \& McNeil (1982).
|
|
|
|
% \subsection{Surrogate AUC Losses}
|
|
% Explain why directly maximizing the empirical AUC is hard: the WMW indicator is pairwise
|
|
% and discontinuous ($O(n^2)$ pairs). Introduce smooth surrogates --- sigmoid approximation
|
|
% to the pairwise indicator, squared loss on score differences --- and their consistency
|
|
% guarantees. Cite Cortes \& Mohri (2003/2004).
|
|
|
|
% \subsection{Stochastic AUC Maximization}
|
|
% Describe mini-batch and compositional stochastic methods that make AUC optimization
|
|
% practical at scale. Cover PESG (Yuan et al., NeurIPS 2020) and LibAUC (Yuan et al.,
|
|
% ICML 2021), which is used as a baseline in \Cref{chapter:experiments}.
|
|
% Key limitation shared by all: they optimize the \emph{whole-curve} AUC, not a specific
|
|
% operating point on the ROC --- motivating the next section.
|
|
|
|
|
|
\subsection{AUC and ROC definitions}
|
|
|
|
A Receiver Operating Characteristic (ROC) is a graph that represents the performance of a classifier.
|
|
The classifier in question is evaluated on binary-labeled data. The labeled data is ordered based on the
|
|
scores given by the classifier. Given any numerical threshold, it is then easy to see which labeled data points fall above and below.
|
|
|
|
For each threshold, the true positive rate and the false positive rate is given. The results for each threshold, and so for each possible TPR and FPR, are put on a graph. The resulting curve is the ROC curve.
|
|
|
|
Formally, the ROC is a set of all points $(FPR(\tau), TPR(\tau))$ where $\tau \in \mathbb{R}$ takes every possible classification threshold value, and $FPR, TPR$ are functions that calculate the relevant statistics at the threshold $\tau$.
|
|
|
|
The AUC is a number in the range $[0, 1]$ representing the total area under the curve. It is calculated by integrating over all TPRs and taking the resulting average FPRs. A result of $1$ represents a perfect classifier,
|
|
while a result of $0$ represents a classifier with a perfect opposite result. A result of $0.5$ represents a classifier that found no separation whatsoever.
|
|
|
|
More formally, the AUC is defined by integrating over FPR and calculating mean TPR:
|
|
|
|
\[
|
|
\text{AUC} = \int_{0}^{1} TPR(FPR^{-1}(\alpha)) d \alpha
|
|
\]
|
|
|
|
Where $TPR$ calculates the TPR at a certain threshold, and $FPR^{-1}(\alpha)$ returns the threshold that gives the requested FPR.
|
|
|
|
It is important to note that, as Wilcoxon-Mann-Whitney showed \cite{hanley1982meaning}, the AUC is equivalent to the probability that the model scores a randomly drawn positive higher than a randomly drawn negative:
|
|
|
|
\[
|
|
\text{AUC} = \mathbb{P}(f(x_1) > f(x_0))
|
|
\]
|
|
|
|
where $x_1 \sim \mathcal{D}_1$ and $x_0 \sim \mathcal{D}_0$ are drawn independently from the positive and negative class distributions, and $f$ is the scoring function.
|
|
|
|
\subsection{AUC as a metric of model performance}
|
|
|
|
The AUC is a direct representation of the performance of a classifier, as opposed to a loss. There are instances where the loss may be minimized, yet separation might be minimal. These two metrics might correlate, but do not necessarily correspond.
|
|
|
|
An example of this is a result for which all true samples receive a score of $0.51$ and all false samples receive a score of $0.49$. Using many losses, including BCE, this would result in a relatively high loss value. On this score however, the AUC is $1$. The model perfectly separates the samples, so further training is not necessary.
|
|
|
|
Direct AUC optimization can be ideal with optimizing for imbalanced datasets. With most losses, such as BCE or focal, the loss is calculated per-sample and then summed (or averaged). In order to properly train for both true and false populations, the scores are weighted, so that equal weight falls on both the true and the false populations. With direct AUC optimization, the imbalance does not affect the calculation, and the reweighting can be forgone.
|
|
|
|
The noise-robustness properties of ranking-based objectives are discussed in \Cref{sec:noisy-labels}.
|
|
|
|
\subsection{AUC optimization in literature}
|
|
|
|
While AUC as a metric for model performance has existed for a while, methods to optimize for it directly are relatively recent. At its core, it is a non-trivial problem to solve. Existing efforts to directly optimize AUC have been based on the Wilcoxon-Mann-Whitney interpretation of AUC. However, directly calculating the aforementioned probability of a random positive score being higher than a negative one introduces two complications. The first is that one must iterate over $n^2$ pairs to cover every possible pair. Additionally, the pairwise function one must evaluate ($\mathbb{P}(f(x_1) > f(x_0))$) is a discontinuous stepwise function, and is therefore non-differentiable.
|
|
|
|
The discontinuous stepwise nature of the evaluated pairwise function is generally addressed by using a surrogate, either sigmoid-like or hinge functions are applied to the score difference~\cite{cortes2003auc}.
|
|
\citet{yuan2021auc} proposed a formulation of the loss that, using specific pairwise surrogate losses, allows training without separately evaluating every pair, but rather, evaluating all given samples once. Their PESG optimizer and the LibAUC library are the primary practical tools for deep AUC maximization and serve as the main AUC baseline in this thesis.
|
|
|
|
% ============================================================
|
|
% PLACEMENT: Section 2.2, after ROC/AUC. Core differentiating
|
|
% claim of this thesis: ROLL's distributional objective is
|
|
% structurally noise-robust without explicit correction.
|
|
% Motivates both the practical results in Ch.\ 4 and the
|
|
% design choices in Ch.\ 3.
|
|
% ============================================================
|
|
\section{Learning with Noisy Labels}
|
|
\label{sec:noisy-labels}
|
|
|
|
While collecting data with correct labels should be a priority, in some cases it is impossible.
|
|
Many data collection methods lead to imperfect labeling, resulting in a muddy ground truth.
|
|
This could be due to a number of factors. The labels might be based on human labeling, which is error prone.
|
|
The labeling might be true, yet the signal might be noisy or wrong, creating a mismatch between signal and label.
|
|
The labeling could be a resource-intensive process, and so it might only be done for a portion of the data.
|
|
And in some cases, the labeling is done based on a different learned model, which too could be faulty.
|
|
|
|
\subsection{Label noise formalization}
|
|
|
|
Given a dataset $ \mathcal{D} = \{(x_i, y_i)\} $, we would define the noisy version as $ \tilde{\mathcal{D}} = \{(x_i, \tilde{y_i})\} $. The noise $N_i$ is defined as:
|
|
|
|
\[
|
|
N_i =
|
|
\begin{cases}
|
|
0 & \text{if } y_i = \tilde{y_i} \\
|
|
1 & \text{otherwise}
|
|
\end{cases}
|
|
\]
|
|
|
|
Label noise is usually categorized as being symmetric, asymmetric or instance dependent.
|
|
|
|
For instance dependent noise, $\mathbb{P}(N_i = 1)$ depends on $x_i$. That is, the probability that a label is flipped depends on $x_i$. For the other two types, the probability of the label being flipped depends only on the original label itself.
|
|
|
|
For symmetric noise, $\mathbb{P}(N_i = 1 | y_i = 1) = \mathbb{P}(N_i = 1 | y_i = 0)$. The probability of the label being flipped does not depend on the original label. This is unlike asymmetric noise, for which the probability of the label flipping does depend on the original label.
|
|
|
|
\subsection{Existing approaches}
|
|
|
|
Prior work has been done in order to make noise-robust learning processes. These can be categorized into three main approaches.
|
|
|
|
\subsubsection{Label meta-training and adjustment}
|
|
Under this approach, the training process is adjusted by modelling label noise explicitly.
|
|
\citet{patrini2017making} introduced forward and backward loss correction: by estimating a
|
|
noise transition matrix $T$, where $T_{ij} = P(\tilde{y}=j \mid y=i)$, the observed noisy
|
|
loss can be corrected to recover the clean-label risk.
|
|
Estimating $T$ requires identifying high-confidence \emph{anchor points} for each class from
|
|
a pre-trained noisy model, a two-phase procedure that may fail under high or asymmetric
|
|
noise rates.
|
|
\citet{ren2018learning} sidestep transition matrix estimation by using a small clean
|
|
validation set and meta-gradient descent to assign per-sample loss weights at each training
|
|
step, adapting dynamically to whatever noise is present without assuming a fixed noise
|
|
structure.
|
|
Both approaches require either prior knowledge of the noise model or access to a clean
|
|
held-out set, which is not always available in practice.
|
|
|
|
\subsubsection{Sample selection}
|
|
Rather than modifying the loss, sample selection methods filter likely-noisy examples during
|
|
training.
|
|
The approach is grounded in the \emph{memorization effect} documented by
|
|
\citet{arpit2017memorization}: deep networks learn clean-label patterns first and memorize
|
|
noisy labels only later in training.
|
|
This means that at early training stages, samples with small loss are more likely to be
|
|
correctly labelled and can be preferentially retained.
|
|
\citet{han2018coteaching} exploit this with Co-teaching: two networks train in parallel,
|
|
each selecting its own small-loss samples and passing them to the other network for the
|
|
parameter update, preventing both from converging to the same memorized errors.
|
|
\citet{li2020dividemix} extend the idea by fitting a per-class Gaussian mixture model to
|
|
the loss distribution at each epoch to separate clean from noisy samples, then treating the
|
|
noisy partition as unlabeled data within a semi-supervised learning framework.
|
|
These methods can achieve strong performance but substantially increase training complexity,
|
|
requiring multiple models or multi-phase procedures.
|
|
|
|
\subsubsection{Robust loss functions}
|
|
A third line of work designs loss functions that are intrinsically resistant to label noise,
|
|
requiring no changes to the training procedure beyond swapping the loss term.
|
|
\citet{ghosh2017robust} established a sufficient condition for noise tolerance: a loss
|
|
$\ell$ tolerates uniform symmetric noise if and only if
|
|
|
|
\[
|
|
\sum_{k=1}^{K} \ell(f(\mathbf{x}), k) = C \quad \text{for all } \mathbf{x},\, f
|
|
\]
|
|
|
|
for some constant $C$ independent of $f(\mathbf{x})$.
|
|
Intuitively, if the total loss over all class assignments is constant, randomly flipping
|
|
labels cannot shift the expected risk minimizer.
|
|
Mean Absolute Error (MAE) satisfies this condition, one can verify that
|
|
$\sum_k (1 - f_k(\mathbf{x})) = K - 1$ for a $K$-class softmax output,
|
|
whereas cross-entropy does not.
|
|
However, MAE converges slowly in practice because its gradient has constant magnitude
|
|
regardless of prediction confidence, making it insensitive to easy examples that should
|
|
receive small updates.
|
|
|
|
\citet{zhang2018generalized} proposed the Generalised Cross Entropy (GCE) loss, defined via
|
|
the $L_q$ family:
|
|
|
|
\[
|
|
\ell_q(f(\mathbf{x}), y) = \frac{1 - f_y(\mathbf{x})^q}{q}, \quad q \in (0,\,1]
|
|
\]
|
|
|
|
where $f_y(\mathbf{x})$ is the model's predicted probability for the true class $y$.
|
|
As $q \to 0$ the loss recovers cross-entropy (via L'H\^{o}pital's rule); at $q = 1$ it
|
|
reduces to MAE, which satisfies the symmetry condition.
|
|
Intermediate values therefore trade noise tolerance for faster convergence.
|
|
We evaluate GCE at $q = 0.7$ as a baseline in the Experiments chapter.
|
|
\citet{zhou2021asymmetric} extend the theoretical framework to asymmetric noise, introducing
|
|
the \emph{asymmetry ratio} to quantify how noise tolerance degrades when flip rates differ
|
|
between classes, and propose an asymmetric loss that applies distinct focusing parameters to
|
|
positive and negative samples: easy negatives are downweighted more aggressively than positives,
|
|
counteracting both gradient dominance from numerous easy negatives in imbalanced settings and
|
|
the increased noise sensitivity that comes with symmetric focusing.
|
|
This asymmetric loss is evaluated as a baseline (\textbf{Asym.}) in \Cref{chapter:experiments}.
|
|
\citet{feng2020can} confirm that standard cross-entropy cannot be made noise-tolerant
|
|
through simple modification, establishing robust losses as a genuinely distinct family
|
|
rather than a patch on existing objectives.
|
|
|
|
% ============================================================
|
|
% PLACEMENT: Section 2.3, after Noisy Labels. Provides the
|
|
% technical KDE background that Ch.\ 3 (sec:roll-kde) builds
|
|
% on directly. End with a forward pointer to sec:roll-kde.
|
|
% ============================================================
|
|
\section{Kernel Density Estimation}
|
|
\label{sec:kde-background}
|
|
\addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{אמידת צפיפות גרעין (KDE)}}
|
|
|
|
% \subsection{The Parzen Window Estimator}
|
|
% Define KDE: $\hat{p}(x) = \frac{1}{nh}\sum_i K\!\bigl(\frac{x-x_i}{h}\bigr)$,
|
|
% where $K$ is a kernel and $h > 0$ is the bandwidth.
|
|
% Present common kernels: Gaussian, Epanechnikov, sigmoid. Discuss how kernel
|
|
% smoothness affects gradient properties. Cite Parzen (1962), Silverman (1986).
|
|
|
|
% \subsection{Bandwidth Selection}
|
|
% Describe Silverman's rule-of-thumb ($h = 1.06\,\hat\sigma\,n^{-1/5}$) as the default.
|
|
% Mention cross-validation as the principled alternative, expensive at training time.
|
|
% In ROLL, bandwidth is fixed (Silverman) or treated as a learnable parameter;
|
|
% details deferred to Ch.\ 3.
|
|
|
|
% \subsection{Differentiability and Use as a Loss Ingredient}
|
|
% Explain why KDE with a smooth kernel is differentiable w.r.t.\ the support points $x_i$:
|
|
% the density at any query is a smooth weighted sum over all support points, so gradients
|
|
% flow back to network parameters. This property is why KDE is suitable as a building
|
|
% block inside a loss function. Forward pointer: \Cref{sec:roll-kde} exploits this to
|
|
% construct ROLL's differentiable threshold estimate from the estimated score distributions.
|
|
|
|
When describing the source distribution for samples, rarely do they conform to some parametric distribution.
|
|
Some data, such as adult heights, follows a Gaussian-like distribution; the occurrence of radioactive decay
|
|
events or car arrivals at an intersection follows a Poisson distribution. For arbitrary real-world data,
|
|
however, no such assumption can be made safely.
|
|
|
|
This motivates Kernel Density Estimation (KDE)~\cite{parzen1962estimation}. Rather than assuming a parametric form, KDE uses the
|
|
empirical sample itself as the model: each observed point $x_i$ contributes a smooth bump, a
|
|
\emph{kernel}, to the estimated density. The assumption is that nearby points are plausibly likely,
|
|
so probability mass is spread locally around each sample. Summing all kernels and normalizing yields a
|
|
smooth PDF that mirrors the shape of the underlying distribution without committing to a parametric family.
|
|
|
|
Formally, let $\{x_i\}_{i=1}^n$ be the observed samples, $K$ a kernel function satisfying
|
|
$\int_{-\infty}^{\infty} K(u)\,du = 1$, and $h > 0$ the \emph{bandwidth} controlling the spread of
|
|
each kernel. Setting the correct bandwidth is important - too low a bandwidth parameter and the KDE becomes too jagged, and the probability anywhere there was no candidate approaches zero, avoiding generalization. Too high a bandwidth parameter, on the other hand, smooths out the distribution entirely, and the entire distribution starts to approach that of the kernel itself, losing the specificity that the candidates bring.
|
|
|
|
The KDE density estimator is:
|
|
\[
|
|
\hat{p}(x) = \frac{1}{nh} \sum_{i=1}^{n} K\!\left(\frac{x - x_i}{h}\right)
|
|
\]
|
|
Each term $\frac{1}{h}K\!\bigl(\frac{x-x_i}{h}\bigr)$ is a kernel centered at $x_i$; the
|
|
$1/h$ factor compensates for the change of variables and ensures the full sum integrates to 1.
|
|
|
|
Valid kernels must be non-negative, symmetric around zero, and normalized. These differ in smoothness, tail behavior, and computational cost. A particularly useful choice is
|
|
the \emph{logistic kernel}:
|
|
\[
|
|
K(u) = \frac{e^{-u}}{(1 + e^{-u})^2}
|
|
\]
|
|
which is smooth, bell-shaped, and centered at zero. Its key property is that its CDF has a closed form,
|
|
which is the sigmoid function:
|
|
\[
|
|
\int_{-\infty}^{x} K(u)\,du \;=\; \frac{1}{1+e^{-x}} \;=:\; \sigma(x)
|
|
\]
|
|
This gives the KDE cumulative distribution function a closed form as well:
|
|
\[
|
|
\hat{F}(\tau) = \frac{1}{n}\sum_{i=1}^{n} \sigma\!\left(\frac{\tau - x_i}{h}\right)
|
|
\]
|
|
|
|
|
|
% ============================================================
|
|
% PLACEMENT: Section 2.4, after KDE. Short motivating context
|
|
% for the TPR@FPR variant (roll-tpr90). Defines the operating-
|
|
% point problem; the next section (NP) gives its theoretical
|
|
% backing. Keep to ~1--2 pages.
|
|
% ============================================================
|
|
\section{Imbalanced Classification}
|
|
\label{sec:imbalanced-tpr-fpr}
|
|
|
|
It is rare for a dataset to have an equal amount of true and false samples. Class imbalance
|
|
is something that must be taken into account during training on most datasets.
|
|
Extreme imbalance exacerbates this issue further - for a dataset with an imbalance ratio of $100$,
|
|
a \emph{constant} classifier (outputting only the most probable class) achieves a very good result.
|
|
Without correction, many learning methods will find a local minima in a relatively trivial solution, such
|
|
as constant output, and have a hard time adjusting. Even if not in this local minima, less emphasis is put
|
|
on the rare class during training, as it contributes less to the loss metric, and the model tends to
|
|
generalize on the rare class poorly.
|
|
|
|
Dealing with this depends on the loss used. For most losses, for which the loss for each sample is calculated separately,
|
|
the solution is to simply weight the losses for each class differently,
|
|
by multiplying the loss for the rare class
|
|
so that it contributes equally. Other approaches using pairwise losses, such as those implemented in LibAUC \cite{yuan2021auc}, naturally weight positive and negative contributions to learning through their implementation.
|
|
|
|
% ============================================================
|
|
% PLACEMENT: Section 2.5, after Imbalanced Classification.
|
|
% Provides the statistical framing for the TPR@FPR objective
|
|
% as the Neyman-Pearson test. Bridges to Ch.\ 3.
|
|
% ============================================================
|
|
|
|
\section{The Neyman-Pearson Lemma}
|
|
\label{sec:neyman-pearson}
|
|
|
|
When maximizing AUC, we are maximizing the mean true-positive-rate over all false-positive-rates.
|
|
When broken down per false-positive rate, it is identical to the problem studied by
|
|
Neyman and Pearson in their 1933 paper
|
|
\cite{neyman1933ix}, as they formulated the tradeoff between the two rates and suggested an
|
|
optimal solution given certain conditions.
|
|
|
|
In the hypothesis-testing formulation of binary classification, a
|
|
sample $\mathbf{x}$ is classified by choosing between $H_0$ (negative
|
|
class, $y=0$) and $H_1$ (positive class, $y=1$). A decision rule
|
|
$\phi\colon \mathcal{X} \to \{0,1\}$ is a \emph{test at level
|
|
$\alpha$} if its false positive rate satisfies
|
|
\[
|
|
P\!\bigl(\phi(\mathbf{x}) = 1 \mid y = 0\bigr) \leq \alpha.
|
|
\]
|
|
This is the probability of it misclassifying a false label, and we say that it does so at a rate no greater
|
|
than $\alpha$.
|
|
|
|
The \emph{power} (true positive rate) is
|
|
$P(\phi(\mathbf{x}) = 1 \mid y = 1)$.
|
|
Among all tests at level $\alpha$, the Neyman-Pearson lemma
|
|
\cite{neyman1933ix} identifies the most effective one (which they cite as the most powerful): that
|
|
which maximizes the \emph{likelihood ratio test}:
|
|
\[
|
|
\phi^*(\mathbf{x})
|
|
= \mathbf{1}\!\left[
|
|
\frac{p(\mathbf{x} \mid y=1)}{p(\mathbf{x} \mid y=0)} > \eta
|
|
\right]
|
|
\]
|
|
Where $\phi^*(\mathbf{x})$ is our theoretical classifier,
|
|
and $\eta \geq 0$ is chosen so that the FPR equals exactly $\alpha$.
|
|
Given these assumptions, and the target FPR that gives $\eta$, the classifier~$\phi^*$ is proven by the lemma to be optimal.
|
|
|
|
\subsection{The practical gap}
|
|
|
|
The NP lemma provides the theoretically optimal solution but assumes the class-conditional densities
|
|
$p(\mathbf{x} \mid y=0)$ and $p(\mathbf{x} \mid y=1)$ are known.
|
|
In practice these densities must be estimated from data.
|
|
A model must be fit to the data in order to estimate these probabilities, meaning that
|
|
there is no guarantee on optimality.
|
|
|
|
Further work regarding \emph{Neyman-Pearson classification} adapts the NP
|
|
principle to the finite-sample setting using nonparametric methods
|
|
\cite{tong2016neyman}.
|
|
These approaches generally split the training and threshold-setting into two
|
|
discrete steps. None of these methods use TPR or FPR optimal threshold as part
|
|
of training for gradient-based learning.
|