Before method rewrite
This commit is contained in:
@@ -170,6 +170,14 @@
|
||||
publisher={Radiological Society of North America}
|
||||
}
|
||||
|
||||
@inproceedings{yuan2021auc,
|
||||
title={Large-scale Robust Deep {AUC} Maximization: A New Surrogate Loss and Empirical Studies on Medical Image Classification},
|
||||
author={Yuan, Zhuoning and Yan, Yan and Sonka, Milan and Yang, Tianbao},
|
||||
booktitle={Proceedings of the {IEEE/CVF} International Conference on Computer Vision ({ICCV})},
|
||||
pages={3040--3049},
|
||||
year={2021}
|
||||
}
|
||||
|
||||
@article{neyman1933ix,
|
||||
title={On the problem of the most efficient tests of statistical hypotheses},
|
||||
author={Neyman, Jerzy and Pearson, Egon S.},
|
||||
|
||||
@@ -1,149 +1,8 @@
|
||||
% ============================================================
|
||||
% PLACEMENT: near the START of Ch. 2, before KDE and AUC/lit
|
||||
% review sections. This is the reader's first encounter with
|
||||
% the problem setting and the TPR@FPR objective. The arc
|
||||
% should be: (1) standard accuracy under imbalance fails;
|
||||
% (2) class-weighting is the usual fix but gives no direct
|
||||
% operating-point control; (3) define TPR, FPR, ROC formally;
|
||||
% (4) state the target: maximise TPR at a fixed FPR budget α.
|
||||
% ============================================================
|
||||
\section{Imbalanced Classification}
|
||||
\label{sec:imbalanced-tpr-fpr}
|
||||
|
||||
% IMPROVEMENT SUGGESTIONS (carried over from original draft):
|
||||
% - Narrative arc is currently backwards. Drive the section
|
||||
% with: accuracy is gameable under imbalance -> class
|
||||
% weighting is the standard fix -> it gives no direct
|
||||
% control over the operating point -> therefore we need
|
||||
% TPR@FPR directly.
|
||||
% - The inseparability tangent (cascade-classifier example)
|
||||
% dilutes the argument; trim to one sentence or cut.
|
||||
% - $\mathcal{D}_0$ and $\mathcal{D}_1$ must be defined
|
||||
% before first use.
|
||||
% - Land the section on a concrete, formal statement of the
|
||||
% TPR@FPR objective with proper notation.
|
||||
|
||||
When fitting a classifier, it is most common to try and maximize the probability
|
||||
of success. That is, given some dataset
|
||||
$\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^{n}$
|
||||
with inputs $\mathbf{x}_i \in \mathcal{X}$ and binary labels $y_i \in \{0, 1\}$, we learn a scoring function $f_\theta : \mathcal{X} \to \mathbb{R}$ and apply a threshold $\tau$ to produce predictions $\hat{y} = \mathbf{1}[f_\theta(\mathbf{x}) > \tau]$. The standard objective is to minimize the probability of classification error:
|
||||
|
||||
$$\mathcal{L}(\theta) = \frac{1}{n} \sum_{i=1}^{n} \mathbf{1}[\hat{y}_i \neq y_i] = P(\hat{y} \neq y)$$
|
||||
|
||||
which is equivalent to maximizing classification accuracy $P(\hat{y} = y)$.
|
||||
|
||||
For most applications, this optimization is warranted. However, there are many real-world settings in which
|
||||
this type of fitting is ill-suited.
|
||||
|
||||
One such application is in the case of inseparable classes. In many cases, no meaningful
|
||||
separation boundary can be found between $\mathcal{D}_0$ and $\mathcal{D}_1$. In many cases,
|
||||
especially in tabular datasets, there can be two data elements $i,j$ for which $\mathbf{x}_i = \mathbf{x}_j$,
|
||||
however, $y_i \neq y_j$. Any deterministic classifier will have to decide whether to classify
|
||||
inseparable points as one class or the other, but it remains hard to control the behavior of
|
||||
a classifier while it is fitting to suggest one outcome or the other.
|
||||
|
||||
The issue of inseparability becomes more apparent in certain applications where runtime is a key
|
||||
and limiting factor. In many classification pipelines, a ``quick and dirty'' classifier may
|
||||
filter out many candidates before an ``in depth'' classifier gives the final answer. This may be done
|
||||
in events where running the full in-depth classification on all data points is not a cost-effective or time-efficient solution.
|
||||
The ``quick and dirty'' classifier can be either low in parameters, making it run quickly. It may also only
|
||||
make a quick decision based on part of the data point. For example, in the case of a classifier looking to flag
|
||||
images of the ocean, a ``quick and dirty'' classifier may look at a very-low resolution version of the image and simply check if it is relatively ``blue''. It filters out many certain false candidates this way, then passes on the remainder to an in-depth, higher accuracy classifier. Solutions like this allow to run at near-optimal performance with lower compute resources in limited-time applications. These situations are common in large-volume image applications on the internet, as well as in astronomy/microscopy.
|
||||
|
||||
Another such application is a case of highly imbalanced data. Even with separability, it is very
|
||||
easy to a classifier to simply ``ignore'' the minority class. This may very likely be a local minimum
|
||||
while fitting, which may be inescapable. As was once said, ``a classifier that always returns 0 is
|
||||
well fitted for flagging pictures of Michael Jordan bathing in green M\&Ms''. Combined with inseparability, we see that in extreme cases, classifiers may choose to ignore, or otherwise penalize, the minority class.
|
||||
|
||||
For every trained classifier, the output per model is a score (TODO rewrite and formalize).
|
||||
|
||||
TODO finish this section
|
||||
|
||||
\begin{itemize}
|
||||
\item What usually happens
|
||||
\begin{itemize}
|
||||
\item Fitting
|
||||
\item Graphing ROC and picking threshold based on tradeoff
|
||||
\end{itemize}
|
||||
\item Usually we compensate for poor performance with
|
||||
\begin{itemize}
|
||||
\item Weighting between classes
|
||||
\item Possible other methods
|
||||
\end{itemize}
|
||||
\item Problem with this method is lack of control.
|
||||
\end{itemize}
|
||||
|
||||
|
||||
% Motivate why standard loss functions (cross-entropy, MSE) break down under class
|
||||
% imbalance: they optimize average accuracy, which a model can game by predicting
|
||||
% the majority class. Introduce concrete examples (e.g. medical diagnosis, fraud
|
||||
% detection) where the cost of a false negative vastly outweighs a false positive,
|
||||
% and where the practitioner needs to operate at a specific FPR budget. Argue that
|
||||
% what is actually needed is direct control over TPR at a fixed FPR, not a
|
||||
% surrogate that only loosely correlates with it.
|
||||
|
||||
|
||||
% ============================================================
|
||||
% PLACEMENT: Section 2.2, immediately after the imbalanced
|
||||
% classification section. Provides the statistical framing
|
||||
% for the TPR@FPR objective — the Neyman-Pearson test — before
|
||||
% surveying the AUC-based practical methods in the next section.
|
||||
% ============================================================
|
||||
\section{Connection to Neyman-Pearson}
|
||||
\label{sec:neyman-pearson}
|
||||
|
||||
The objective introduced in \Cref{sec:imbalanced-tpr-fpr},
|
||||
maximizing the true positive rate subject to a false positive rate
|
||||
budget $\alpha$, is precisely the problem studied by Neyman and
|
||||
Pearson in their 1933 paper \cite{neyman1933ix}, as they formulated
|
||||
the tradeoff between the two rates and suggested the existence of the
|
||||
optimal solution.
|
||||
|
||||
\subsection{The Neyman-Pearson Lemma}
|
||||
|
||||
In the hypothesis-testing formulation of binary classification, a
|
||||
sample $\mathbf{x}$ is classified by choosing between $H_0$ (negative
|
||||
class, $y=0$) and $H_1$ (positive class, $y=1$). A decision rule
|
||||
$\phi\colon \mathcal{X} \to \{0,1\}$ is a \emph{test at level
|
||||
$\alpha$} if its false positive rate satisfies
|
||||
\[
|
||||
P\!\bigl(\phi(\mathbf{x}) = 1 \mid y = 0\bigr) \leq \alpha.
|
||||
\]
|
||||
This is the probability of it misclassifying a false label, and we say that it does so at a rate no greater
|
||||
than $\alpha$.
|
||||
|
||||
The \emph{power} (true positive rate) is
|
||||
$P(\phi(\mathbf{x}) = 1 \mid y = 1)$.
|
||||
Among all tests at level $\alpha$, the Neyman-Pearson lemma
|
||||
\cite{neyman1933ix} identifies the most effective one (which they site as the most powerful): that
|
||||
which maximizes the \emph{likelyhood ratio test}:
|
||||
\[
|
||||
\phi^*(\mathbf{x})
|
||||
= \mathbf{1}\!\left[
|
||||
\frac{p(\mathbf{x} \mid y=1)}{p(\mathbf{x} \mid y=0)} > \eta
|
||||
\right]
|
||||
\]
|
||||
Where $\phi^*(\mathbf{x})$ is our theoretical classifier,
|
||||
and $\eta \geq 0$ is chosen so that the FPR equals exactly $\alpha$.
|
||||
No other test at level $\alpha$ achieves higher power.
|
||||
|
||||
\subsection{The practical gap}
|
||||
|
||||
We note thta this optimal classifier can only be derived for datasets for which this likelyhood can be derived.
|
||||
On datasets with a finite number of possible data points (for example, datasets with purely categorical features), and assuming the datasets is purely representative, one could create such a classifier. However importantly, this cannot be done for real datasets. For datasets with scalar dimensions, one cannot identify such a probability for each candidate, since every candidate probably appears only once, and so has an assigned probability per class of either $1$ or $0$.
|
||||
|
||||
Further work regarding \emph{Neyman-Pearson classification} adapts the NP
|
||||
principle to the finite-sample setting using nonparametric methods
|
||||
\cite{tong2016neyman}.
|
||||
These approaches generally split the training and threshold-setting into two
|
||||
discrete steps. None of these methods use TPR or FPR optimal threshold as part
|
||||
of training for gradient-based learning.
|
||||
|
||||
% ============================================================
|
||||
% PLACEMENT: Section 2.3, after the Neyman-Pearson section.
|
||||
% Establishes AUC as the standard practical measure and
|
||||
% surveys how the literature optimizes it, showing that
|
||||
% whole-curve methods do not provide single-point control.
|
||||
% PLACEMENT: Section 2.1 — opens Chapter 2. Establishes AUC
|
||||
% as the standard measure for ranking-based classification and
|
||||
% surveys how the literature optimizes it directly. Sets up
|
||||
% the noise robustness discussion in the next section.
|
||||
% ============================================================
|
||||
\section{ROC Curves and AUC Optimization}
|
||||
\label{sec:roc-auc}
|
||||
@@ -215,191 +74,19 @@ The noise-robustness properties of ranking-based objectives are discussed in \Cr
|
||||
|
||||
While AUC as a metric for model performance has existed for a while, methods to optimize for it directly are relatively recent. At it's core, it is a non-trivial problem to solve. Existing efforts to directly optimize AUC have been based on the Wilcoxon-Mann-Whitney interpretation of AUC. However, directly calculating the afformentioned probability of a random positive score behing higher than a negative one introduces two complications. The first is that one must iterate over $n^2$ pairs to cover every posible pair. Additionally, the pairwise function one must evaluate ( $\mathbb{P}(f(x_1) > f(x_0))$ ) is a non-contiguous stepwise function, and is therefor on-differentiable.
|
||||
|
||||
The non-contiguous stepwise nature of the evaluated pairwise function is generally fixed by using a surrogate - either sigmoid-like or hinge functions are applied to the score difference.
|
||||
%TODO add references
|
||||
|
||||
The non-contiguous stepwise nature of the evaluated pairwise function is generally fixed by using a surrogate — either sigmoid-like or hinge functions are applied to the score difference.
|
||||
\citet{yuan2021auc} proposed a compositional stochastic gradient framework that reformulates AUC maximization as a min-max saddle-point problem, enabling mini-batch training without maintaining all pairwise scores in memory. Their PESG optimizer and the LibAUC library are the primary practical tools for deep AUC maximization and serve as the main AUC baseline in this thesis.
|
||||
|
||||
% ============================================================
|
||||
% PLACEMENT: Section 2.4, after the ROC/AUC section and before
|
||||
% the noisy labels section. Provides the technical KDE background
|
||||
% that Ch.\ 3 (sec:roll-kde) builds on directly. End with a
|
||||
% forward pointer to sec:roll-kde.
|
||||
% ============================================================
|
||||
\section{Kernel Density Estimation}
|
||||
\label{sec:kde-background}
|
||||
\addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{אמידת צפיפות גרעין (KDE)}}
|
||||
|
||||
% \subsection{The Parzen Window Estimator}
|
||||
% Define KDE: $\hat{p}(x) = \frac{1}{nh}\sum_i K\!\bigl(\frac{x-x_i}{h}\bigr)$,
|
||||
% where $K$ is a kernel and $h > 0$ is the bandwidth.
|
||||
% Present common kernels: Gaussian, Epanechnikov, sigmoid. Discuss how kernel
|
||||
% smoothness affects gradient properties. Cite Parzen (1962), Silverman (1986).
|
||||
|
||||
% \subsection{Bandwidth Selection}
|
||||
% Describe Silverman's rule-of-thumb ($h = 1.06\,\hat\sigma\,n^{-1/5}$) as the default.
|
||||
% Mention cross-validation as the principled alternative, expensive at training time.
|
||||
% In ROLL, bandwidth is fixed (Silverman) or treated as a learnable parameter;
|
||||
% details deferred to Ch.\ 3.
|
||||
|
||||
% \subsection{Differentiability and Use as a Loss Ingredient}
|
||||
% Explain why KDE with a smooth kernel is differentiable w.r.t.\ the support points $x_i$:
|
||||
% the density at any query is a smooth weighted sum over all support points, so gradients
|
||||
% flow back to network parameters. This property is why KDE is suitable as a building
|
||||
% block inside a loss function. Forward pointer: \Cref{sec:roll-kde} exploits this to
|
||||
% construct ROLL's differentiable threshold estimate from the estimated score distributions.
|
||||
|
||||
When describing the source distribution for samples, rarely do they conform to some parametric distribution.
|
||||
Some data, such as adult heights, follows a Gaussian-like distribution; the occurrence of radioactive decay
|
||||
events or car arrivals at an intersection follows a Poisson distribution. For arbitrary real-world data,
|
||||
however, no such assumption can be made safely.
|
||||
|
||||
This motivates Kernel Density Estimation (KDE). Rather than assuming a parametric form, KDE uses the
|
||||
empirical sample itself as the model: each observed point $x_i$ contributes a smooth bump --- a
|
||||
\emph{kernel} --- to the estimated density. The assumption is that nearby points are plausibly likely,
|
||||
so probability mass is spread locally around each sample. Summing all kernels and normalizing yields a
|
||||
smooth PDF that mirrors the shape of the underlying distribution without committing to a parametric family.
|
||||
|
||||
Formally, let $\{x_i\}_{i=1}^n$ be the observed samples, $K$ a kernel function satisfying
|
||||
$\int_{-\infty}^{\infty} K(u)\,du = 1$, and $h > 0$ the \emph{bandwidth} controlling the spread of
|
||||
each kernel. Setting the correct bandwidth is important - too low a bandwidth parameter and the KDE becomes too jagged, and the probability anywhere there was no candidate approaches zero, avoiding generalization. Too high a bandwidth parameter, on the other hand, smooths out the distribution entirely, and the entire distribution starts to approach that of the kernel itself, losing the specificity that the candidates bring.
|
||||
|
||||
The KDE density estimator is:
|
||||
\[
|
||||
\hat{p}(x) = \frac{1}{nh} \sum_{i=1}^{n} K\!\left(\frac{x - x_i}{h}\right)
|
||||
\]
|
||||
Each term $\frac{1}{h}K\!\bigl(\frac{x-x_i}{h}\bigr)$ is a kernel centered at $x_i$; the
|
||||
$1/h$ factor compensates for the change of variables and ensures the full sum integrates to 1.
|
||||
|
||||
Valid kernels must be non-negative, symmetric around zero, and normalized. These differ in smoothness, tail behavior, and computational cost. A particularly useful choice is
|
||||
the \emph{logistic kernel}:
|
||||
\[
|
||||
K(u) = \frac{e^{-u}}{(1 + e^{-u})^2}
|
||||
\]
|
||||
which is smooth, bell-shaped, and centered at zero. Its key property is that its CDF has a closed form,
|
||||
which is the sigmoid function:
|
||||
\[
|
||||
\int_{-\infty}^{x} K(u)\,du \;=\; \frac{1}{1+e^{-x}} \;=:\; \sigma(x)
|
||||
\]
|
||||
This gives the KDE cumulative distribution function a closed form as well:
|
||||
\[
|
||||
\hat{F}(\tau) = \frac{1}{n}\sum_{i=1}^{n} \sigma\!\left(\frac{\tau - x_i}{h}\right)
|
||||
\]
|
||||
|
||||
|
||||
% ============================================================
|
||||
% PLACEMENT: Section 2.5, AFTER the KDE background section
|
||||
% (sec:kde-background) and BEFORE the Neyman-Pearson bridge
|
||||
% (sec:neyman-pearson, which becomes 2.6).
|
||||
%
|
||||
% Purpose: establish that existing pointwise losses are
|
||||
% fragile under label corruption, motivating both the
|
||||
% practical robustness of ROLL and the poisoning experiment
|
||||
% results in Ch. 4.
|
||||
%
|
||||
% Narrative arc:
|
||||
% 1. Define label noise types: symmetric (uniform flip),
|
||||
% asymmetric / class-conditional (only certain class
|
||||
% transitions occur), instance-dependent. Our poisoning
|
||||
% experiment is asymmetric: P(\tilde{y}=0 \mid y=1)>0,
|
||||
% P(\tilde{y}=1 \mid y=0)=0.
|
||||
% 2. Why BCE is fragile: the symmetry condition (Ghosh
|
||||
% et al., AAAI 2017) — a loss \ell is noise-tolerant
|
||||
% iff \sum_k \ell(f(x),k) = C (constant). CE fails
|
||||
% this; MAE satisfies it but converges slowly. Cite
|
||||
% Feng et al. (IJCAI 2020) for "Can CE be robust?"
|
||||
% (only under very strict conditions). Implication:
|
||||
% mislabeled samples produce large unmitigated
|
||||
% gradients; DNNs memorize noise late in training
|
||||
% (memorization effect).
|
||||
% 3. Family of existing fixes — keep brief; these are what
|
||||
% ROLL sidesteps structurally:
|
||||
% (a) Robust loss functions: MAE (noise-tolerant,
|
||||
% slow), GCE (Zhang & Sabuncu, NeurIPS 2018;
|
||||
% interpolates MAE↔CE via q), asymmetric losses
|
||||
% (Zhou et al., ICML 2021), symmetrization
|
||||
% (arXiv:2605.20347).
|
||||
% (b) Label correction via noise transition matrix
|
||||
% (Patrini et al., CVPR 2017): estimate
|
||||
% T[i,j]=P(\tilde{y}=j|y=i), correct logits by
|
||||
% T^{-1}. Hard for asymmetric noise without a
|
||||
% clean reference set.
|
||||
% (c) Sample selection / co-teaching (Han et al.,
|
||||
% NeurIPS 2018): two networks select small-loss
|
||||
% examples for each other, exploiting the
|
||||
% memorization effect. Requires dual-model
|
||||
% training.
|
||||
% 4. Ranking / distributional objectives and noise
|
||||
% robustness: pairwise losses depend on score
|
||||
% differences, not absolute labels — one mislabeled
|
||||
% sample affects O(n) pairs, each with a small gradient
|
||||
% weight. Wang et al. (arXiv:2305.14258) formally show
|
||||
% noise robustness of partial-AUC objectives.
|
||||
% 5. Bridge (observation, not oversell): ROLL's KDE-based
|
||||
% distributional objective shares this structural
|
||||
% property. Fake negatives (copies of positives with
|
||||
% flipped labels) shift the estimated negative density
|
||||
% but do not erase distributional separation when true
|
||||
% positive scores remain clustered. ROLL achieves this
|
||||
% without any explicit noise-handling mechanism; the
|
||||
% poisoning experiment in Ch. 4 provides empirical
|
||||
% evidence.
|
||||
% PLACEMENT: Section 2.2, after ROC/AUC. Core differentiating
|
||||
% claim of this thesis: ROLL's distributional objective is
|
||||
% structurally noise-robust without explicit correction.
|
||||
% Motivates both the practical results in Ch.\ 4 and the
|
||||
% design choices in Ch.\ 3.
|
||||
% ============================================================
|
||||
\section{Learning with Noisy Labels}
|
||||
\label{sec:noisy-labels}
|
||||
|
||||
% SUBSECTION STRUCTURE:
|
||||
%
|
||||
% \subsection{Types of Label Noise}
|
||||
% - Symmetric / uniform: each label flipped with
|
||||
% probability \varepsilon across all classes.
|
||||
% - Asymmetric / class-conditional: only certain class
|
||||
% transitions occur. Our poisoning experiment is of
|
||||
% this type: P(\tilde{y}=0 \mid y=1)>0,
|
||||
% P(\tilde{y}=1 \mid y=0)=0.
|
||||
% - Instance-dependent: flip probability varies per
|
||||
% sample (hardest to handle; mention only briefly).
|
||||
% - Note: even moderate noise rates significantly hurt
|
||||
% pointwise losses in the imbalanced setting, because
|
||||
% mislabeled positives are rare to begin with.
|
||||
%
|
||||
% \subsection{Why Cross-Entropy Is Fragile Under Label Noise}
|
||||
% - Introduce symmetry condition \cite{ghosh2017robust}:
|
||||
% a loss is noise-tolerant iff
|
||||
% \sum_k \ell(f(\mathbf{x}), k) = C (constant).
|
||||
% - CE does NOT satisfy this; MAE does (but is slow).
|
||||
% - Memorization effect: DNNs learn clean patterns early,
|
||||
% then memorize noisy labels late in training.
|
||||
% - Cite: \cite{ghosh2017robust}, \cite{feng2020can}.
|
||||
%
|
||||
% \subsection{Approaches to Robust Learning}
|
||||
% - Paragraph 1 — Robust loss functions:
|
||||
% MAE \cite{ghosh2017robust},
|
||||
% GCE \cite{zhang2018generalized} (L_q interpolation),
|
||||
% asymmetric losses \cite{zhou2021asymmetric}.
|
||||
% Key tradeoff: noise-tolerance vs.\ convergence speed.
|
||||
% - Paragraph 2 — Transition matrix correction
|
||||
% \cite{patrini2017making}: forward correction
|
||||
% multiplies logits by T^{-1}. Canonical method but
|
||||
% requires estimating T, which is hard for asymmetric
|
||||
% noise without a clean anchor set.
|
||||
% - Paragraph 3 — Sample selection / co-teaching
|
||||
% \cite{han2018coteaching}: two nets select small-loss
|
||||
% samples for each other each epoch. Bypasses T
|
||||
% estimation but requires dual-model training.
|
||||
%
|
||||
% \subsection{Ranking Objectives and Noise Robustness}
|
||||
% - Pairwise ranking losses are structurally more
|
||||
% noise-robust: a single mislabeled sample affects O(n)
|
||||
% pairs, each with small gradient weight, vs.\ one large
|
||||
% pointwise gradient in BCE.
|
||||
% - Cite Wang et al.\ \cite{wang2023weakly} for formal
|
||||
% treatment in the partial-AUC setting.
|
||||
% - Close with bridge sentence: ROLL's distributional
|
||||
% threshold objective operates in the same structural
|
||||
% regime — not pairwise, but also not pointwise —
|
||||
% conferring noise robustness without explicit correction.
|
||||
% Empirical evidence deferred to the Experiments chapter.
|
||||
|
||||
While collecting data with correct labels should be a priority, in some cases it is impossible.
|
||||
Many data collection methods lead to imperfect labeling, resulting in a muddy ground truth.
|
||||
This could be due to a number of factors. The labels might be based on human labeling, which is error prone.
|
||||
@@ -435,7 +122,7 @@ Under this approach, the training process is adjusted by modelling label noise e
|
||||
noise transition matrix $T$, where $T_{ij} = P(\tilde{y}=j \mid y=i)$, the observed noisy
|
||||
loss can be corrected to recover the clean-label risk.
|
||||
Estimating $T$ requires identifying high-confidence \emph{anchor points} for each class from
|
||||
a pre-trained noisy model --- a two-phase procedure that may fail under high or asymmetric
|
||||
a pre-trained noisy model, a two-phase procedure that may fail under high or asymmetric
|
||||
noise rates.
|
||||
\citet{ren2018learning} sidestep transition matrix estimation by using a small clean
|
||||
validation set and meta-gradient descent to assign per-sample loss weights at each training
|
||||
@@ -474,8 +161,8 @@ $\ell$ tolerates uniform symmetric noise if and only if
|
||||
for some constant $C$ independent of $f(\mathbf{x})$.
|
||||
Intuitively, if the total loss over all class assignments is constant, randomly flipping
|
||||
labels cannot shift the expected risk minimizer.
|
||||
Mean Absolute Error (MAE) satisfies this condition --- one can verify that
|
||||
$\sum_k (1 - f_k(\mathbf{x})) = K - 1$ for a $K$-class softmax output ---
|
||||
Mean Absolute Error (MAE) satisfies this condition, one can verify that
|
||||
$\sum_k (1 - f_k(\mathbf{x})) = K - 1$ for a $K$-class softmax output,
|
||||
whereas cross-entropy does not.
|
||||
However, MAE converges slowly in practice because its gradient has constant magnitude
|
||||
regardless of prediction confidence, making it insensitive to easy examples that should
|
||||
@@ -500,13 +187,146 @@ between classes.
|
||||
through simple modification, establishing robust losses as a genuinely distinct family
|
||||
rather than a patch on existing objectives.
|
||||
|
||||
All three families above treat noise as a quantity to be explicitly corrected --- by
|
||||
adjusting labels, filtering samples, or redesigning the loss.
|
||||
ROLL takes a different route: its distributional threshold objective aggregates over the
|
||||
full score distribution of each class, so a small fraction of mislabelled samples shifts
|
||||
the estimated density slightly but cannot dominate the threshold.
|
||||
This structural robustness arises from the distributional formulation itself, without any
|
||||
explicit noise-handling mechanism; empirical evidence is provided in the Experiments
|
||||
chapter.
|
||||
% ============================================================
|
||||
% PLACEMENT: Section 2.3, after Noisy Labels. Provides the
|
||||
% technical KDE background that Ch.\ 3 (sec:roll-kde) builds
|
||||
% on directly. End with a forward pointer to sec:roll-kde.
|
||||
% ============================================================
|
||||
\section{Kernel Density Estimation}
|
||||
\label{sec:kde-background}
|
||||
\addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{אמידת צפיפות גרעין (KDE)}}
|
||||
|
||||
% \subsection{The Parzen Window Estimator}
|
||||
% Define KDE: $\hat{p}(x) = \frac{1}{nh}\sum_i K\!\bigl(\frac{x-x_i}{h}\bigr)$,
|
||||
% where $K$ is a kernel and $h > 0$ is the bandwidth.
|
||||
% Present common kernels: Gaussian, Epanechnikov, sigmoid. Discuss how kernel
|
||||
% smoothness affects gradient properties. Cite Parzen (1962), Silverman (1986).
|
||||
|
||||
% \subsection{Bandwidth Selection}
|
||||
% Describe Silverman's rule-of-thumb ($h = 1.06\,\hat\sigma\,n^{-1/5}$) as the default.
|
||||
% Mention cross-validation as the principled alternative, expensive at training time.
|
||||
% In ROLL, bandwidth is fixed (Silverman) or treated as a learnable parameter;
|
||||
% details deferred to Ch.\ 3.
|
||||
|
||||
% \subsection{Differentiability and Use as a Loss Ingredient}
|
||||
% Explain why KDE with a smooth kernel is differentiable w.r.t.\ the support points $x_i$:
|
||||
% the density at any query is a smooth weighted sum over all support points, so gradients
|
||||
% flow back to network parameters. This property is why KDE is suitable as a building
|
||||
% block inside a loss function. Forward pointer: \Cref{sec:roll-kde} exploits this to
|
||||
% construct ROLL's differentiable threshold estimate from the estimated score distributions.
|
||||
|
||||
When describing the source distribution for samples, rarely do they conform to some parametric distribution.
|
||||
Some data, such as adult heights, follows a Gaussian-like distribution; the occurrence of radioactive decay
|
||||
events or car arrivals at an intersection follows a Poisson distribution. For arbitrary real-world data,
|
||||
however, no such assumption can be made safely.
|
||||
|
||||
This motivates Kernel Density Estimation (KDE). Rather than assuming a parametric form, KDE uses the
|
||||
empirical sample itself as the model: each observed point $x_i$ contributes a smooth bump, a
|
||||
\emph{kernel}, to the estimated density. The assumption is that nearby points are plausibly likely,
|
||||
so probability mass is spread locally around each sample. Summing all kernels and normalizing yields a
|
||||
smooth PDF that mirrors the shape of the underlying distribution without committing to a parametric family.
|
||||
|
||||
Formally, let $\{x_i\}_{i=1}^n$ be the observed samples, $K$ a kernel function satisfying
|
||||
$\int_{-\infty}^{\infty} K(u)\,du = 1$, and $h > 0$ the \emph{bandwidth} controlling the spread of
|
||||
each kernel. Setting the correct bandwidth is important - too low a bandwidth parameter and the KDE becomes too jagged, and the probability anywhere there was no candidate approaches zero, avoiding generalization. Too high a bandwidth parameter, on the other hand, smooths out the distribution entirely, and the entire distribution starts to approach that of the kernel itself, losing the specificity that the candidates bring.
|
||||
|
||||
The KDE density estimator is:
|
||||
\[
|
||||
\hat{p}(x) = \frac{1}{nh} \sum_{i=1}^{n} K\!\left(\frac{x - x_i}{h}\right)
|
||||
\]
|
||||
Each term $\frac{1}{h}K\!\bigl(\frac{x-x_i}{h}\bigr)$ is a kernel centered at $x_i$; the
|
||||
$1/h$ factor compensates for the change of variables and ensures the full sum integrates to 1.
|
||||
|
||||
Valid kernels must be non-negative, symmetric around zero, and normalized. These differ in smoothness, tail behavior, and computational cost. A particularly useful choice is
|
||||
the \emph{logistic kernel}:
|
||||
\[
|
||||
K(u) = \frac{e^{-u}}{(1 + e^{-u})^2}
|
||||
\]
|
||||
which is smooth, bell-shaped, and centered at zero. Its key property is that its CDF has a closed form,
|
||||
which is the sigmoid function:
|
||||
\[
|
||||
\int_{-\infty}^{x} K(u)\,du \;=\; \frac{1}{1+e^{-x}} \;=:\; \sigma(x)
|
||||
\]
|
||||
This gives the KDE cumulative distribution function a closed form as well:
|
||||
\[
|
||||
\hat{F}(\tau) = \frac{1}{n}\sum_{i=1}^{n} \sigma\!\left(\frac{\tau - x_i}{h}\right)
|
||||
\]
|
||||
|
||||
|
||||
% ============================================================
|
||||
% PLACEMENT: Section 2.4, after KDE. Short motivating context
|
||||
% for the TPR@FPR variant (roll-tpr90). Defines the operating-
|
||||
% point problem; the next section (NP) gives its theoretical
|
||||
% backing. Keep to ~1--2 pages.
|
||||
% ============================================================
|
||||
\section{Imbalanced Classification}
|
||||
\label{sec:imbalanced-tpr-fpr}
|
||||
|
||||
It is rare for a dataset to have an equal amount of true and false samples. Class imbalance
|
||||
is something that must be taken into account during training on most datasets.
|
||||
Extreme imbalance exacerbates this issue further - for a dataset with an imbalance ratio of $100$,
|
||||
a \emph{constant} classifier (outputting only the most probable class) acheives a very good result.
|
||||
Without correction, many learning methods will find a local minima in a relatively trivial solution, such
|
||||
as constant output, and have a hard time adjusting. Even if not in this local minima, less emphasis is put
|
||||
on the rare class during training, as it contributes less to the loss metric, and the model tends to
|
||||
generalize on the rare class poorly.
|
||||
|
||||
Dealing with this depends on the loss used. For most losses, for which the loss for each sample is calculated seperately,
|
||||
is to simply weight the losses differently, by multiplying the loss for the rare class
|
||||
so that it contributes equally. Other approaches using pairwise losses, such as those implemented in LibAUC \cite{yuan2021auc}, naturally weight positive and negative contributions to learning through their implementation.
|
||||
|
||||
% ============================================================
|
||||
% PLACEMENT: Section 2.5, after Imbalanced Classification.
|
||||
% Provides the statistical framing for the TPR@FPR objective
|
||||
% as the Neyman-Pearson test. Bridges to Ch.\ 3.
|
||||
% ============================================================
|
||||
|
||||
\section{The Neyman-Pearson Lemma}
|
||||
\label{sec:neyman-pearson}
|
||||
|
||||
When maximizing AUC, we are maximizing the mean true-positive-rate over all false-positive-rates.
|
||||
When broken down per false-positive rate, it is identical to the problem studied by
|
||||
Neyman and Pearson in their 1933 paper
|
||||
\cite{neyman1933ix}, as they formulated the tradeoff between the two rates and suggested an
|
||||
optimal solution given certain conditions.
|
||||
|
||||
In the hypothesis-testing formulation of binary classification, a
|
||||
sample $\mathbf{x}$ is classified by choosing between $H_0$ (negative
|
||||
class, $y=0$) and $H_1$ (positive class, $y=1$). A decision rule
|
||||
$\phi\colon \mathcal{X} \to \{0,1\}$ is a \emph{test at level
|
||||
$\alpha$} if its false positive rate satisfies
|
||||
\[
|
||||
P\!\bigl(\phi(\mathbf{x}) = 1 \mid y = 0\bigr) \leq \alpha.
|
||||
\]
|
||||
This is the probability of it misclassifying a false label, and we say that it does so at a rate no greater
|
||||
than $\alpha$.
|
||||
|
||||
The \emph{power} (true positive rate) is
|
||||
$P(\phi(\mathbf{x}) = 1 \mid y = 1)$.
|
||||
Among all tests at level $\alpha$, the Neyman-Pearson lemma
|
||||
\cite{neyman1933ix} identifies the most effective one (which they cite as the most powerful): that
|
||||
which maximizes the \emph{likelihood ratio test}:
|
||||
\[
|
||||
\phi^*(\mathbf{x})
|
||||
= \mathbf{1}\!\left[
|
||||
\frac{p(\mathbf{x} \mid y=1)}{p(\mathbf{x} \mid y=0)} > \eta
|
||||
\right]
|
||||
\]
|
||||
Where $\phi^*(\mathbf{x})$ is our theoretical classifier,
|
||||
and $\eta \geq 0$ is chosen so that the FPR equals exactly $\alpha$.
|
||||
Given these assumptions, and the target FPR that gives $\eta$, the classifier$\phi^*$ is proven by the lemma to be optimal.
|
||||
|
||||
\subsection{The practical gap}
|
||||
|
||||
The NP lemma provides the theoretically optimal solution but assumes the class-conditional densities
|
||||
$p(\mathbf{x} \mid y=0)$ and $p(\mathbf{x} \mid y=1)$ are known.
|
||||
In practice these densities must be estimated from data.
|
||||
A model must be fit to the data in order to estimate these probabilities, meaning that
|
||||
there is no gaurantee on optimality.
|
||||
|
||||
Further work regarding \emph{Neyman-Pearson classification} adapts the NP
|
||||
principle to the finite-sample setting using nonparametric methods
|
||||
\cite{tong2016neyman}.
|
||||
These approaches generally split the training and threshold-setting into two
|
||||
discrete steps. None of these methods use TPR or FPR optimal threshold as part
|
||||
of training for gradient-based learning.
|
||||
|
||||
Reference in New Issue
Block a user