Interim
This commit is contained in:
@@ -158,3 +158,24 @@
|
||||
pages={12846--12856},
|
||||
year={2021}
|
||||
}
|
||||
|
||||
@article{hanley1982meaning,
|
||||
title={The meaning and use of the area under a receiver operating characteristic ({ROC}) curve},
|
||||
author={Hanley, James A and McNeil, Barbara J},
|
||||
journal={Radiology},
|
||||
volume={143},
|
||||
number={1},
|
||||
pages={29--36},
|
||||
year={1982},
|
||||
publisher={Radiological Society of North America}
|
||||
}
|
||||
|
||||
@article{neyman1933ix,
|
||||
title={On the problem of the most efficient tests of statistical hypotheses},
|
||||
author={Neyman, Jerzy and Pearson, Egon S.},
|
||||
journal={Philosophical Transactions of the Royal Society of London. Series {A}},
|
||||
volume={231},
|
||||
pages={289--337},
|
||||
year={1933},
|
||||
publisher={The Royal Society}
|
||||
}
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
% operating-point control; (3) define TPR, FPR, ROC formally;
|
||||
% (4) state the target: maximise TPR at a fixed FPR budget α.
|
||||
% ============================================================
|
||||
\section{Imbalanced Classification and The TPR/FPR Objective}
|
||||
\section{Imbalanced Classification}
|
||||
\label{sec:imbalanced-tpr-fpr}
|
||||
|
||||
% IMPROVEMENT SUGGESTIONS (carried over from original draft):
|
||||
@@ -85,10 +85,65 @@ TODO finish this section
|
||||
|
||||
% ============================================================
|
||||
% PLACEMENT: Section 2.2, immediately after the imbalanced
|
||||
% classification section (sec:imbalanced-tpr-fpr). Establishes
|
||||
% AUC as the standard measure and surveys how the literature
|
||||
% optimizes it, before narrowing to partial AUC / single-point
|
||||
% objectives in the next section.
|
||||
% classification section. Provides the statistical framing
|
||||
% for the TPR@FPR objective — the Neyman-Pearson test — before
|
||||
% surveying the AUC-based practical methods in the next section.
|
||||
% ============================================================
|
||||
\section{Connection to Neyman-Pearson}
|
||||
\label{sec:neyman-pearson}
|
||||
|
||||
The objective introduced in \Cref{sec:imbalanced-tpr-fpr},
|
||||
maximizing the true positive rate subject to a false positive rate
|
||||
budget $\alpha$, is precisely the problem studied by Neyman and
|
||||
Pearson in their 1933 paper \cite{neyman1933ix}, as they formulated
|
||||
the tradeoff between the two rates and suggested the existence of the
|
||||
optimal solution.
|
||||
|
||||
\subsection{The Neyman-Pearson Lemma}
|
||||
|
||||
In the hypothesis-testing formulation of binary classification, a
|
||||
sample $\mathbf{x}$ is classified by choosing between $H_0$ (negative
|
||||
class, $y=0$) and $H_1$ (positive class, $y=1$). A decision rule
|
||||
$\phi\colon \mathcal{X} \to \{0,1\}$ is a \emph{test at level
|
||||
$\alpha$} if its false positive rate satisfies
|
||||
\[
|
||||
P\!\bigl(\phi(\mathbf{x}) = 1 \mid y = 0\bigr) \leq \alpha.
|
||||
\]
|
||||
This is the probability of it misclassifying a false label, and we say that it does so at a rate no greater
|
||||
than $\alpha$.
|
||||
|
||||
The \emph{power} (true positive rate) is
|
||||
$P(\phi(\mathbf{x}) = 1 \mid y = 1)$.
|
||||
Among all tests at level $\alpha$, the Neyman-Pearson lemma
|
||||
\cite{neyman1933ix} identifies the most effective one (which they site as the most powerful): that
|
||||
which maximizes the \emph{likelyhood ratio test}:
|
||||
\[
|
||||
\phi^*(\mathbf{x})
|
||||
= \mathbf{1}\!\left[
|
||||
\frac{p(\mathbf{x} \mid y=1)}{p(\mathbf{x} \mid y=0)} > \eta
|
||||
\right]
|
||||
\]
|
||||
Where $\phi^*(\mathbf{x})$ is our theoretical classifier,
|
||||
and $\eta \geq 0$ is chosen so that the FPR equals exactly $\alpha$.
|
||||
No other test at level $\alpha$ achieves higher power.
|
||||
|
||||
\subsection{The practical gap}
|
||||
|
||||
We note thta this optimal classifier can only be derived for datasets for which this likelyhood can be derived.
|
||||
On datasets with a finite number of possible data points (for example, datasets with purely categorical features), and assuming the datasets is purely representative, one could create such a classifier. However importantly, this cannot be done for real datasets. For datasets with scalar dimensions, one cannot identify such a probability for each candidate, since every candidate probably appears only once, and so has an assigned probability per class of either $1$ or $0$.
|
||||
|
||||
Further work regarding \emph{Neyman-Pearson classification} adapts the NP
|
||||
principle to the finite-sample setting using nonparametric methods
|
||||
\cite{tong2016neyman}.
|
||||
These approaches generally split the training and threshold-setting into two
|
||||
discrete steps. None of these methods use TPR or FPR optimal threshold as part
|
||||
of training for gradient-based learning.
|
||||
|
||||
% ============================================================
|
||||
% PLACEMENT: Section 2.3, after the Neyman-Pearson section.
|
||||
% Establishes AUC as the standard practical measure and
|
||||
% surveys how the literature optimizes it, showing that
|
||||
% whole-curve methods do not provide single-point control.
|
||||
% ============================================================
|
||||
\section{ROC Curves and AUC Optimization}
|
||||
\label{sec:roc-auc}
|
||||
@@ -125,9 +180,27 @@ scores given by the classifier. Given any numerical threshold, it is then easy t
|
||||
|
||||
For each threshold, the true positive rate and the false positive rate is given. The results for each threshold, and so for each possible TPR and FPR, are put on a graph. The resulting curve is the ROC curve.
|
||||
|
||||
Formally, the ROC is a set of all points $(FPR(\tau), TPR(\tau))$ where $\tau \in \mathbb{R}$ takes every possible classification threshold value, and $FPR, TPR$ are functions that calculate the relevant statistics at the threshold $\tau$.
|
||||
|
||||
The AUC is a number in the range $[0, 1]$ representing the total area under the curve. It is calculated by integrating over all TPRs and taking the resulting average FPRs. A result of $1$ represents a perfect classifier,
|
||||
while a result of $0$ represents a classifier with a perfect opposite result. A result of $0.5$ represents a classifier that found no separation whatsoever.
|
||||
|
||||
More formally, the AUC is defined by integrating over FPR and calculating mean TPR:
|
||||
|
||||
\[
|
||||
\text{AUC} = \int_{0}^{1} TPR(FPR^{-1}(\alpha)) d \alpha
|
||||
\]
|
||||
|
||||
Where $TPR$ calculates the TPR at a certain threshold, and $FPR^{-1}(\alpha)$ returns the threshold that gives the requested FPR.
|
||||
|
||||
It is important to note that, as Wilcoxon-Mann-Whitney showed \cite{hanley1982meaning}, the AUC is equivalent to the probability that the model scores a randomly drawn positive higher than a randomly drawn negative:
|
||||
|
||||
\[
|
||||
\text{AUC} = \mathbb{P}(f(x_1) > f(x_0))
|
||||
\]
|
||||
|
||||
where $x_1 \sim \mathcal{D}_1$ and $x_0 \sim \mathcal{D}_0$ are drawn independently from the positive and negative class distributions, and $f$ is the scoring function.
|
||||
|
||||
\subsection{AUC as a metric of model performance}
|
||||
|
||||
The AUC is a direct representation of the performance of a classifier, as opposed to a loss. There are instances where the loss may be minimized, yet seperation might be minimal. These two metrics might correlate, but do not necessarily correspond.
|
||||
@@ -136,47 +209,19 @@ An example of this is a result for which all true samples receive a score of $0.
|
||||
|
||||
Direct AUC optimization can be ideal with optimizing for imbalanced datasets. With most losses such as BCE or focal, the loss is calculated per-sample and then summed (or averaged). In order to properly train for both true and false populations, the scores are weighted, so that equal weight falls on both the true and the false populations. With direct AUC optimization, the imbalance does not affect the calculation, and the reweighting can be forgone.
|
||||
|
||||
Additionally, direct AUC methods are more resilient to label noise, as opposed to classical methods. Due to inseperability, correctly classifying a sample might require a model to incorrectly classify similar or even identical samples that were mislabeled. This is only worth it for the model in training if the total loss of the correctly labeled samples outweighs the total loss of the incorrectly labeled samples. For most methods, the weight per class is given by the class imbalance in the dataset, but this doesn't take into account the label noise in any way. Optimizing for AUC specifically would eliminate this risk.
|
||||
The noise-robustness properties of ranking-based objectives are discussed in \Cref{sec:noisy-labels}.
|
||||
|
||||
\subsection{AUC optimization in literature}
|
||||
|
||||
While AUC as a metric for model performance has existed for a while, methods to optimize for it directly are relatively recent.
|
||||
While AUC as a metric for model performance has existed for a while, methods to optimize for it directly are relatively recent. At it's core, it is a non-trivial problem to solve. Existing efforts to directly optimize AUC have been based on the Wilcoxon-Mann-Whitney interpretation of AUC. However, directly calculating the afformentioned probability of a random positive score behing higher than a negative one introduces two complications. The first is that one must iterate over $n^2$ pairs to cover every posible pair. Additionally, the pairwise function one must evaluate ( $\mathbb{P}(f(x_1) > f(x_0))$ ) is a non-contiguous stepwise function, and is therefor on-differentiable.
|
||||
|
||||
The non-contiguous stepwise nature of the evaluated pairwise function is generally fixed by using a surrogate - either sigmoid-like or hinge functions are applied to the score difference.
|
||||
%TODO add references
|
||||
|
||||
% ============================================================
|
||||
% PLACEMENT: Section 2.3, after ROC/AUC section above and
|
||||
% before the KDE background section. Narrows from whole-curve
|
||||
% AUC to the single operating-point objective that ROLL targets.
|
||||
% ============================================================
|
||||
\section{Partial AUC and TPR at Fixed FPR}
|
||||
\label{sec:partial-auc}
|
||||
\addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{AUC חלקי ו-TPR בנקודת FPR קבועה}}
|
||||
|
||||
% \subsection{Partial AUC}
|
||||
% Define partial AUC as the area under the ROC curve restricted to $\mathrm{FPR} \in [0, \alpha]$.
|
||||
% Motivate: in high-stakes settings (fraud, medical triage) only the low-FPR region matters.
|
||||
% Survey recent pAUC optimizers that extend the PESG/compositional framework
|
||||
% (Lin et al., 2022; Yang \& Ying, 2022).
|
||||
|
||||
% \subsection{TPR at a Fixed FPR: The Single-Point Objective}
|
||||
% Distinguish pAUC (integral over an interval) from the exact single-point objective:
|
||||
% maximize $\mathrm{TPR}(f, \alpha)$ for a fixed budget $\alpha$.
|
||||
% Argue that post-hoc threshold search and pAUC integration both fail to provide a
|
||||
% differentiable end-to-end training signal for this precise target --- this is the gap
|
||||
% that ROLL fills.
|
||||
|
||||
% \subsection{Neyman-Pearson Classification (Brief)}
|
||||
% Introduce NP classification (Cannon et al., 2002; Tong, 2013) as the statistical
|
||||
% framing: minimize type-II error subject to $\mathrm{FPR} \le \alpha$.
|
||||
% Note that classical NP methods are not gradient-based and cannot be applied to DNNs.
|
||||
% Full treatment deferred to \Cref{sec:neyman-pearson} at the end of this chapter.
|
||||
|
||||
TODO write this section.
|
||||
|
||||
|
||||
% ============================================================
|
||||
% PLACEMENT: Section 2.4, after partial AUC and before the
|
||||
% noisy labels section. Provides the technical KDE background
|
||||
% PLACEMENT: Section 2.4, after the ROC/AUC section and before
|
||||
% the noisy labels section. Provides the technical KDE background
|
||||
% that Ch.\ 3 (sec:roll-kde) builds on directly. End with a
|
||||
% forward pointer to sec:roll-kde.
|
||||
% ============================================================
|
||||
@@ -464,28 +509,4 @@ This structural robustness arises from the distributional formulation itself, wi
|
||||
explicit noise-handling mechanism; empirical evidence is provided in the Experiments
|
||||
chapter.
|
||||
|
||||
% ============================================================
|
||||
% PLACEMENT: Section 2.6 (was 2.5), near the END of Ch. 2,
|
||||
% after KDE and Noisy Labels sections, just before the
|
||||
% chapter summary. This section bridges from prior work to
|
||||
% the method chapter by framing ROLL as a learnable,
|
||||
% gradient-based NP test.
|
||||
% NOTE: the label sec:neyman-pearson is unchanged; existing
|
||||
% \Cref references in Ch. 3 (sec:problem-formulation) still
|
||||
% work without modification.
|
||||
% ============================================================
|
||||
\section{Connection to Neyman-Pearson}
|
||||
\label{sec:neyman-pearson}
|
||||
|
||||
% Introduce the Neyman-Pearson lemma: among all tests at a
|
||||
% given false-positive rate, the likelihood-ratio test
|
||||
% maximizes TPR. Frame ROLL as a learnable, gradient-based
|
||||
% realization of this principle — rather than assuming known
|
||||
% class-conditional distributions (as classical NP does),
|
||||
% ROLL estimates them from model scores during training.
|
||||
% Briefly note that existing NP-inspired methods do not
|
||||
% support deep learning / gradient-based optimization; this
|
||||
% is the gap ROLL fills.
|
||||
|
||||
TODO write this section.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user