Results overhaul
This commit is contained in:
+94
-80
@@ -3,10 +3,10 @@
|
||||
\label{sec:problem-formulation}
|
||||
|
||||
% Motivation for this section — why accuracy fails under class imbalance, why
|
||||
% TPR@FPR gives direct operating-point control, and the Neyman-Pearson framing
|
||||
% that underpins ROLL — is developed in \Cref{chapter:related_work}
|
||||
% (\Cref{sec:imbalanced-tpr-fpr,sec:neyman-pearson}). This section establishes
|
||||
% only the notation and objective used throughout the chapter.
|
||||
% AUC is the natural objective under label noise and imbalance, and the
|
||||
% Neyman-Pearson framing for single-threshold optimization — is developed in
|
||||
% \Cref{chapter:related_work} (\Cref{sec:imbalanced-tpr-fpr,sec:neyman-pearson}).
|
||||
% This section establishes the notation and primary objective used throughout the chapter.
|
||||
Given a dataset $\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^{n}$ with inputs
|
||||
$\mathbf{x}_i \in \mathcal{X}$ and binary labels $y_i \in \{0, 1\}$, we learn a
|
||||
scoring function $f_\theta : \mathcal{X} \to \mathbb{R}$ by fitting.
|
||||
@@ -17,50 +17,51 @@ The positive and negative subsets of $\mathcal{D}$ are:
|
||||
\mathcal{D}_0 = \{(\mathbf{x}_i, y_i) \in \mathcal{D} \mid y_i = 0\}.
|
||||
\]
|
||||
Applying a threshold $\tau$ produces predictions $\hat{y} = \mathbf{1}[f_\theta(\mathbf{x}) > \tau]$.
|
||||
The \emph{true positive rate} (TPR) and \emph{false positive rate} (FPR) at $\tau$ are:
|
||||
The \emph{true positive rate} (TPR) and \emph{false positive rate} (FPR) at threshold $\tau$ are:
|
||||
\begin{align}
|
||||
\operatorname{TPR}(\alpha) &= P\!\bigl(f_\theta(\mathbf{x}) > \tau \mid y = 1\bigr), \\
|
||||
\operatorname{FPR}(\alpha) &= P\!\bigl(f_\theta(\mathbf{x}) > \tau \mid y = 0\bigr).
|
||||
\operatorname{TPR}(\tau) &= P\!\bigl(f_\theta(\mathbf{x}) > \tau \mid y = 1\bigr), \\
|
||||
\operatorname{FPR}(\tau) &= P\!\bigl(f_\theta(\mathbf{x}) > \tau \mid y = 0\bigr).
|
||||
\end{align}
|
||||
In our scenario, we either wish to learn $f_{\theta}$ in order to maximize $\operatorname{TPR}$
|
||||
subject to a fixed $\operatorname{FPR}$, or vice versa. Our fixed $\operatorname{TPR}$ or $\operatorname{FPR}$
|
||||
will be fixed to a value $\alpha \in (0, 1)$.
|
||||
|
||||
Therefor, when maximizing our $\operatorname{TPR}$ subject to a fixed $\operatorname{FPR}$, we are solving for:
|
||||
For $\alpha \in (0,1)$, let $\tau^*(\alpha)$ denote the threshold achieving
|
||||
$\operatorname{FPR}(\tau^*(\alpha)) = \alpha$, and write
|
||||
$\operatorname{TPR}(\alpha) = \operatorname{TPR}(\tau^*(\alpha))$ for the true positive rate
|
||||
at that operating point. The curve $\alpha \mapsto \operatorname{TPR}(\alpha)$ is the ROC curve.
|
||||
The primary learning objective is to maximize its area:
|
||||
\begin{equation}
|
||||
\max_\theta \; \operatorname{TPR}\!\bigl(\tau^*(\theta)\bigr)
|
||||
\quad \text{s.t.} \quad \operatorname{FPR}\!\bigl(\tau^*(\theta)\bigr) = \alpha,
|
||||
\label{eq:objective}
|
||||
\max_\theta\;\operatorname{AUC}(f_\theta) = \int_0^1 \operatorname{TPR}(\alpha)\,d\alpha.
|
||||
\label{eq:objective-auc}
|
||||
\end{equation}
|
||||
Maximizing AUC is the natural objective under class imbalance and label noise, where no
|
||||
single operating threshold is fixed in advance.
|
||||
|
||||
where $\tau^*(\theta)$ is the threshold that achieves exactly $\operatorname{FPR} = \alpha$
|
||||
for the current model $f_\theta$.
|
||||
Conversley, when we are minimizing $\operatorname{FPR}$ subject to a fixed $\operatorname{TPR}$ we are solving for:
|
||||
|
||||
\begin{equation}
|
||||
\min_\theta \; \operatorname{FPR}\!\bigl(\tau^*(\theta)\bigr)
|
||||
\quad \text{s.t.} \quad \operatorname{TPR}\!\bigl(\tau^*(\theta)\bigr) = \alpha,
|
||||
\label{eq:objective}
|
||||
\end{equation}
|
||||
|
||||
The ROLL framework, introduced in
|
||||
\Cref{sec:roll-framework}, provides a method of differentiable surrogate for these objectives.
|
||||
As a special case, one may instead optimize a single operating point: maximize
|
||||
$\operatorname{TPR}$ at a fixed $\operatorname{FPR} = \alpha_0$, or minimize
|
||||
$\operatorname{FPR}$ at a fixed $\operatorname{TPR} = \alpha_0$.
|
||||
The ROLL framework, introduced in \Cref{sec:roll-framework}, provides differentiable
|
||||
surrogates for both the AUC objective \eqref{eq:objective-auc} and these point objectives.
|
||||
|
||||
%------------------------------------------------
|
||||
|
||||
\section{The ROLL Framework}
|
||||
\label{sec:roll-framework}
|
||||
|
||||
In order to allow gradient-based learning methods to optimize TPR or FPR directly, we introduce
|
||||
ROLL - Rate Optimized Likelyhood Loss.
|
||||
In order to allow gradient-based learning methods to optimize AUC directly, we introduce
|
||||
ROLL --- Rate Optimized Likelihood Loss.
|
||||
|
||||
The core idea of ROLL is as follows: when training, the user selects that they would like to optimize
|
||||
TPR given some fixed FPR. Then, during training, on every training step, the actual TPR is calculated for
|
||||
the given FPR, and the fitting algorithm tries to maximize the TPR directly. This way, while training, we gain direct control over the performance at the threshold that we care about - avoiding needing to pick out a threshold later.
|
||||
The core idea of ROLL is as follows: at each training step, differentiable distributions
|
||||
are fitted to the positive- and negative-class score outputs of $f_\theta$. AUC is then
|
||||
approximated by evaluating the estimated $\operatorname{TPR}$ at 99 evenly-spaced
|
||||
$\operatorname{FPR}$ values across $(0,1)$ and averaging --- a numerical integration of
|
||||
the ROC curve. Because each per-threshold evaluation is differentiable through the fitted
|
||||
distributions, this AUC approximation can be maximized directly by gradient descent, with
|
||||
no threshold selection required at inference time.
|
||||
|
||||
In Neyman-Pearson terms~\cite{tong2016neyman}, our algorithm wishes to directly minimize the Type-1 error given a fixed Type-2 error, or vice versa.
|
||||
ROLL also supports single-threshold optimization: selecting a target
|
||||
$\operatorname{FPR} = \alpha_0$ and maximizing $\operatorname{TPR}$ at exactly that point.
|
||||
In Neyman-Pearson terms~\cite{tong2016neyman}, this corresponds to minimizing Type-I error
|
||||
at a fixed Type-II error bound.
|
||||
|
||||
In practice, however, things are not so simple.
|
||||
In practice, however, making these evaluations differentiable is not straightforward.
|
||||
|
||||
\subsection{The Differentiability Problem}
|
||||
\label{sec:differentiability-problem}
|
||||
@@ -116,15 +117,14 @@ every step. Contiguous probability distributions are contiguous w.r.t their defi
|
||||
Additionally, we can calculate the TPR, FPR given the probability distributions CDF and ICDF functions. If we can gaurantee these functions to be differentiable, which proved to be the main challenge of this work, we can gaurantee the entire process to be differentiable. Once we do so, a derivative of the scores themselves w.r.t. the loss can be acheived.
|
||||
|
||||
|
||||
\subsection{General ROLL Formulation and Derivation}
|
||||
\subsection{Per-Threshold ROLL Loss}
|
||||
\label{sec:roll-formulation}
|
||||
|
||||
% Present the abstract ROLL loss in terms of a generic distribution $\mathcal{F}$
|
||||
% fitted to model scores, without yet committing to a specific family. Derive the
|
||||
% expression for TPR@FPR$_0$ as a function of $\mathcal{F}$'s parameters, and show
|
||||
% that minimizing the loss is equivalent to maximizing TPR subject to the FPR
|
||||
% constraint. Include the full derivation here; this is the core theoretical
|
||||
% contribution of the chapter.
|
||||
% expression for TPR@FPR as a function of $\mathcal{F}$'s parameters. This
|
||||
% per-threshold loss is the building block that ROLL-AOC averages over 99 FPR values
|
||||
% to obtain the primary AUC objective.
|
||||
|
||||
Using the dataset $\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^{n}$ as introduced in
|
||||
\Cref{sec:problem-formulation}, let $\mathcal{B} \subseteq \mathcal{D}$ denote a training batch
|
||||
@@ -255,49 +255,38 @@ Unifying both cases, the general gradient applicable to all ROLL instantiations
|
||||
The concrete form of each partial derivative depends on the choice of distribution family
|
||||
$\mathcal{F}$, and is derived for each instantiation in \Cref{sec:roll-instantiations}.
|
||||
|
||||
\subsection{ROLL variant - direct AUC optimisation}
|
||||
|
||||
In addition to optimising for a specific rate, ROLL provides us an opportunity to
|
||||
optimize the AUC of the ROC directly.
|
||||
|
||||
The AUC is defined as the area under the curve of the ROC. The ROC is a function that
|
||||
for every FPR gives us the performing TPR of the model we have fitted.
|
||||
|
||||
Formally, for the function $TPR(\alpha)$ as defined above, we may define AUC as:
|
||||
\subsection{ROLL-AOC: AUC Optimization}
|
||||
\label{sec:roll-aoc}
|
||||
|
||||
The primary objective of ROLL is to maximize AUC directly, using the per-threshold
|
||||
loss \eqref{eq:roll-tpr-at-fpr} as a building block. Recall from
|
||||
\Cref{sec:problem-formulation} that AUC is the integral of $\operatorname{TPR}(\alpha)$
|
||||
over all FPR values:
|
||||
\begin{equation}
|
||||
\operatorname{AUC} = \int_{0}^{1}\operatorname{TPR}(\alpha) d \alpha
|
||||
\operatorname{AUC} = \int_{0}^{1}\operatorname{TPR}(\alpha)\,d\alpha.
|
||||
\end{equation}
|
||||
|
||||
The AUC is something we would like to maximize. In order to do so, we would like to
|
||||
minimize the area over the curve, which is $1-AUC$.
|
||||
|
||||
We note that $1-\operatorname{TPR}(\alpha) = \mathcal{L}(TPR@FPR)$.
|
||||
|
||||
From here, we would like to minimize:
|
||||
|
||||
Maximizing AUC is equivalent to minimizing the \emph{Area Over the Curve} (AOC),
|
||||
$1 - \operatorname{AUC}$. Since $1 - \operatorname{TPR}(\alpha) =
|
||||
\mathcal{L}_{\text{ROLL-TPR@FPR}}(\alpha)$ by definition \eqref{eq:roll-tpr-at-fpr}:
|
||||
\begin{equation}
|
||||
\mathcal{L}_{AOC} = \int_0^1\mathcal{L}_{\text{ROLL-TPR@FPR}}(\alpha) d \alpha
|
||||
\mathcal{L}_{\text{AOC}} = \int_0^1\mathcal{L}_{\text{ROLL-TPR@FPR}}(\alpha)\,d\alpha.
|
||||
\end{equation}
|
||||
|
||||
A direct integration is not part of this work. However, integration can be approximated. Instead
|
||||
of integrating directly, it is possible to evaluate the loss for every FPR between 0 and 1, sum them, and re-normalize.
|
||||
|
||||
Therefor, subdividing the interval $(0, 1)$ to 99 steps (every alpha starting from $0.01$ and ending with $0.99$, since a value of $0$ or $1$ would be illegal), our loss for minimizing AOC becomes:
|
||||
|
||||
Direct integration is not tractable, but the integral is well-approximated by a Riemann
|
||||
sum. Evaluating at 99 evenly-spaced FPR values $\alpha = i/100$ for $i = 1, \ldots, 99$
|
||||
(excluding the degenerate endpoints 0 and 1), the ROLL-AOC loss is:
|
||||
\begin{equation}
|
||||
\mathcal{L}_{ROLL-AOC} = \frac{1}{99}\sum_{i=1}^{99}\mathcal{L}_{ROLL-TPR@FPR}\left(\frac{i}{100}\right)
|
||||
\mathcal{L}_{\text{ROLL-AOC}} = \frac{1}{99}\sum_{i=1}^{99}\mathcal{L}_{\text{ROLL-TPR@FPR}}\!\left(\frac{i}{100}\right)
|
||||
\label{eq:roll-aoc}
|
||||
\end{equation}
|
||||
|
||||
The gradient per score then simply becomes the average gradient over every threshold.
|
||||
|
||||
The gradient follows by linearity of the sum:
|
||||
\begin{equation}
|
||||
\frac{\partial \mathcal{L}_{ROLL-AOC}(f_\theta(\mathbf{X}))}
|
||||
{\partial f_\theta(\mathbf{x}_i)} = \frac{1}{99}\sum_{i=1}^{99}\left(\frac{\partial \mathcal{L}_{\text{ROLL-TPR@FPR}}\left(f_\theta(\mathbf{X}) ; \frac{i}{100}\right)}
|
||||
{\partial f_\theta(\mathbf{x}_i)} \right)
|
||||
\frac{\partial \mathcal{L}_{\text{ROLL-AOC}}(f_\theta(\mathbf{X}))}
|
||||
{\partial f_\theta(\mathbf{x}_i)} = \frac{1}{99}\sum_{i=1}^{99}\frac{\partial \mathcal{L}_{\text{ROLL-TPR@FPR}}\!\left(f_\theta(\mathbf{X}) ; \frac{i}{100}\right)}
|
||||
{\partial f_\theta(\mathbf{x}_i)}
|
||||
\label{eq:roll-aoc-gradient}
|
||||
\end{equation}
|
||||
The concrete per-threshold gradient used in the sum is derived for each distribution
|
||||
family in \Cref{sec:roll-instantiations}.
|
||||
|
||||
|
||||
%------------------------------------------------
|
||||
@@ -327,6 +316,8 @@ While the distributions of scores cannot be gauranteed to follow any form, there
|
||||
some instances where the score distributions might be similar or exactly gaussian.
|
||||
For example, when finding linear separability between gaussian distributions, the resulting
|
||||
scores will be distributed normally, and this method will fit exactly.
|
||||
In ROLL-AOC, the Gaussian instantiation evaluates \eqref{eq:roll-tpr-at-fpr-gaussian}
|
||||
at each of the 99 FPR grid points and averages the results.
|
||||
|
||||
\subsubsection{Forward}
|
||||
\label{sec:roll-gaussian-forward}
|
||||
@@ -401,6 +392,9 @@ Beta ROLL applies when model scores are constrained to the open interval $(0,1)$
|
||||
as is the case when $f_\theta$ uses a sigmoid output activation. The Beta distribution
|
||||
is supported on $(0,1)$ and its two shape parameters allow it to capture a wide range
|
||||
of unimodal and skewed score distributions that a Gaussian cannot represent.
|
||||
In ROLL-AOC, Beta ROLL evaluates \eqref{eq:roll-tpr-at-fpr-beta} at each of the 99
|
||||
FPR grid points and averages the results, with gradients propagated via automatic
|
||||
differentiation.
|
||||
|
||||
\subsubsection{Forward}
|
||||
\label{sec:roll-beta-forward}
|
||||
@@ -470,6 +464,9 @@ are therefore propagated via automatic differentiation through the estimator
|
||||
Unfortunately, the score output of trained models remains unpredictable. A solution for this is to use a Kernel Density Estimation (KDE) for probability
|
||||
estimation. This benefits from being far more adaptable to real-world distributions of outputs of our trained models. The tradeoff of course is computational
|
||||
complexity - computing the gradient derivation in such a case is non-trivial.
|
||||
KDE ROLL is the primary instantiation used in ROLL-AOC: its non-parametric flexibility
|
||||
makes it the most accurate estimator of the ROC curve when score distributions are
|
||||
unknown, at the cost of evaluating the custom backward pass 99 times per training step.
|
||||
|
||||
\subsubsection{Forward}
|
||||
\label{sec:roll-kde-forward}
|
||||
@@ -632,13 +629,20 @@ The Beta distribution only accepts values in the interval $\[0, 1\]$. Since it
|
||||
is mapped only over an interval, the translation-invariance assumption does not hold
|
||||
for this distribution.
|
||||
|
||||
\begin{corollary}[Gradient balance under ROLL-AOC]
|
||||
\label{cor:grad-balance-aoc}
|
||||
$\mathcal{L}_{\emph{\text{ROLL-AOC}}}$ also satisfies gradient balance: since it is a uniform
|
||||
average of 99 per-threshold objectives each satisfying \eqref{eq:grad-balance}, the property
|
||||
is preserved by linearity of summation.
|
||||
\end{corollary}
|
||||
|
||||
\paragraph{Gradient locality.}
|
||||
\label{para:gradient-locality}
|
||||
|
||||
A second structural consequence of the ROLL-KDE formulation is that gradient mass is
|
||||
\emph{concentrated near the operating threshold $\tau$}. Samples whose scores lie
|
||||
close to $\tau$ receive larger gradient updates, while samples
|
||||
already far on the correct side contribute very little.
|
||||
A second structural consequence of the ROLL-KDE formulation, when optimizing at a
|
||||
single operating threshold, is that gradient mass is \emph{concentrated near that
|
||||
threshold $\tau$}. Samples whose scores lie close to $\tau$ receive larger gradient
|
||||
updates, while samples already far on the correct side contribute very little.
|
||||
|
||||
This follows directly from \Cref{eq:kde-grad-combined}. For the positive class ($y_i = 1$),
|
||||
the gradient magnitude is
|
||||
@@ -663,12 +667,12 @@ This stands in contrast to standard losses such as cross-entropy, where the per-
|
||||
gradient is determined by the predicted probability, with no direct dependence on an
|
||||
operating threshold.
|
||||
|
||||
\begin{observation}[Gradient locality]
|
||||
\begin{observation}[Gradient locality, single-threshold]
|
||||
\label{obs:gradient-locality}
|
||||
For KDE-ROLL, the gradient magnitude of sample $\mathbf{x}_i$ is proportional to
|
||||
the kernel PDF evaluated at the distance between $f_\theta(\mathbf{x}_i)$ and the
|
||||
operating threshold $\tau$. Samples nearest to $\tau$ receive the largest updates;
|
||||
samples far from $\tau$ in either direction contribute negligible gradient.
|
||||
For KDE-ROLL at a single operating threshold, the gradient magnitude of sample
|
||||
$\mathbf{x}_i$ is proportional to the kernel PDF evaluated at the distance between
|
||||
$f_\theta(\mathbf{x}_i)$ and $\tau$. Samples nearest to $\tau$ receive the largest
|
||||
updates; samples far from $\tau$ in either direction contribute negligible gradient.
|
||||
\end{observation}
|
||||
|
||||
\Cref{fig:threshold-locality} illustrates this for a simulated two-class setting. The
|
||||
@@ -689,6 +693,16 @@ concentration of gradient mass around $\tau$, with smooth decay on either side.
|
||||
\label{fig:threshold-locality}
|
||||
\end{figure}
|
||||
|
||||
\begin{remark}[Gradient coverage under ROLL-AOC]
|
||||
\label{rem:aoc-coverage}
|
||||
\Cref{obs:gradient-locality} describes gradient concentration at a single threshold.
|
||||
Under ROLL-AOC, 99 distinct thresholds spanning $\operatorname{FPR} \in \{0.01,\ldots,0.99\}$
|
||||
are active simultaneously, so the gradient is a superposition of 99 bell-shaped kernels
|
||||
spread across the full score range. This broadens gradient coverage relative to
|
||||
single-threshold optimization, making ROLL-AOC more suitable when no specific operating
|
||||
point is fixed in advance.
|
||||
\end{remark}
|
||||
|
||||
%------------------------------------------------
|
||||
|
||||
\section{Implementation Considerations}
|
||||
|
||||
Reference in New Issue
Block a user