Updated results, hebrew, abstract, intro, etc.
This commit is contained in:
+31
-52
@@ -73,7 +73,7 @@ In practice, however, making these evaluations differentiable is not straightfor
|
||||
% common surrogates (AUC proxies, hinge-based losses) only indirectly target
|
||||
% the operating point a practitioner cares about.
|
||||
|
||||
Directly optimizing TPR for FPR, or vice versa, is an indefferentiable problem.
|
||||
Directly optimizing TPR for FPR, or vice versa, is a non-differentiable problem.
|
||||
|
||||
For this sub-section, we will reffer to an example of maximizing TPR given a fixed FPR,
|
||||
though this can be reversed.
|
||||
@@ -87,7 +87,7 @@ There are two inherent steps built-in to our methods:
|
||||
|
||||
\begin{itemize}
|
||||
\item Based on fixed rate, calculate threshold that gives said rate.
|
||||
For example, when fixing FPR = $\alpha$, find some threshold $\tau$ s.t. $\mathbb{E}[1_{f_{\theta}\mathbf{x}_i > \tau | y_i = 0}] = \alpha$
|
||||
For example, when fixing FPR = $\alpha$, find some threshold $\tau$ such that $\mathbb{E}[\mathbf{1}[f_\theta(\mathbf{x}_i) > \tau \mid y_i = 0]] = \alpha$
|
||||
\item Given the calculated threshold, calculate the rate we would like to maximize
|
||||
(in our example, maximize TPR)
|
||||
\end{itemize}
|
||||
@@ -113,8 +113,8 @@ This necessitates a different approach, to make these calculations differentiabl
|
||||
|
||||
The way to overcome this problem in differentiability is not to calculate the TPR and FPR
|
||||
directly, but to estimate them by fitting probability distributions to the score output of the model at
|
||||
every step. Contiguous probability distributions are contiguous w.r.t their defining parameters, and a derivation of the parameters w.r.t the distributions themselves can be made.
|
||||
Additionally, we can calculate the TPR, FPR given the probability distributions CDF and ICDF functions. If we can gaurantee these functions to be differentiable, which proved to be the main challenge of this work, we can gaurantee the entire process to be differentiable. Once we do so, a derivative of the scores themselves w.r.t. the loss can be acheived.
|
||||
every step. Continuous probability distributions are continuous with respect to their defining parameters, and a derivation of the parameters with respect to the distributions themselves can be made.
|
||||
Additionally, we can calculate the TPR, FPR given the probability distributions CDF and ICDF functions. If we can guarantee these functions to be differentiable, which proved to be the main challenge of this work, we can guarantee the entire process to be differentiable. Once we do so, a derivative of the scores themselves with respect to the loss can be achieved.
|
||||
|
||||
|
||||
\subsection{Per-Threshold ROLL Loss}
|
||||
@@ -163,7 +163,7 @@ Using this notation, at a threshold $\tau$:
|
||||
\begin{align}
|
||||
\operatorname{TPR}(\tau) &= 1 - \hat{F}_1(\tau), \label{eq:tpr-cdf} \\
|
||||
\operatorname{FPR}(\tau) &= 1 - \hat{F}_0(\tau). \label{eq:fpr-cdf}
|
||||
\end{align}t
|
||||
\end{align}
|
||||
|
||||
Let $\hat{F}_1^{-1}$ and $\hat{F}_0^{-1}$ denote the quantile functions of $P_{\hat{\phi}_1}$ and
|
||||
$P_{\hat{\phi}_0}$, respectively. The threshold achieving a target TPR of $\alpha$ follows directly
|
||||
@@ -273,8 +273,7 @@ $1 - \operatorname{AUC}$. Since $1 - \operatorname{TPR}(\alpha) =
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{AOC}} = \int_0^1\mathcal{L}_{\text{ROLL-TPR@FPR}}(\alpha)\,d\alpha.
|
||||
\end{equation}
|
||||
Direct integration is not tractable, but the integral is well-approximated by a Riemann
|
||||
sum. Evaluating at 99 evenly-spaced FPR values $\alpha = i/100$ for $i = 1, \ldots, 99$
|
||||
Direct integration is not tractable, but the integral is well-approximated by averaging out sampled points. Evaluating at 99 evenly-spaced FPR values $\alpha = i/100$ for $i = 1, \ldots, 99$
|
||||
(excluding the degenerate endpoints 0 and 1), the ROLL-AOC loss is:
|
||||
\begin{equation}
|
||||
\mathcal{L}_{\text{ROLL-AOC}} = \frac{1}{99}\sum_{i=1}^{99}\mathcal{L}_{\text{ROLL-TPR@FPR}}\!\left(\frac{i}{100}\right)
|
||||
@@ -291,6 +290,25 @@ The concrete per-threshold gradient for the KDE instantiation is derived in
|
||||
\Cref{sec:roll-instantiations}; gradients for the parametric cases are in
|
||||
\Cref{appendix:gauss-grad,appendix:beta-roll}.
|
||||
|
||||
\Cref{fig:roll-aoc-roc} gives a geometric view of this objective.
|
||||
|
||||
\begin{figure}[H]
|
||||
\centering
|
||||
\includegraphics[width=0.62\textwidth]{content/method/figures/roll_aoc_roc.pdf}
|
||||
\caption{Geometric interpretation of ROLL-AOC. The ROC curve (orange) maps
|
||||
each FPR value $\alpha$ to $\operatorname{TPR}(\alpha)$.
|
||||
\textbf{Green}: area under the curve (AUC), the quantity we maximize.
|
||||
\textbf{Red}: area over the curve
|
||||
($\mathrm{AOC} = 1 - \mathrm{AUC} = \mathcal{L}_{\text{ROLL-AOC}}$),
|
||||
the loss we minimize.
|
||||
Dots mark 10 evenly-spaced FPR sample points $\alpha_i$; vertical
|
||||
segments show the per-threshold loss contribution
|
||||
$1 - \operatorname{TPR}(\alpha_i) = \mathcal{L}_{\text{ROLL-TPR@FPR}}(\alpha_i)$
|
||||
at each point. Averaging these contributions (99 in practice, 10 shown
|
||||
for clarity) gives the value of \eqref{eq:roll-aoc}.}
|
||||
\label{fig:roll-aoc-roc}
|
||||
\end{figure}
|
||||
|
||||
|
||||
%------------------------------------------------
|
||||
|
||||
@@ -336,7 +354,7 @@ As before, to calculate the loss, we calculate:
|
||||
\mathcal{L}_{\text{ROLL-TPR@FPR}}^{\text{KDE}}(f_{\theta}(\mathcal{B}) ; \alpha) = \hat{F}_1(\hat{F}_0^{-1}(1-\alpha))
|
||||
\]
|
||||
|
||||
To calculate this, we first define the kernel function as $\sigma '$, and the CDF of the kernel function as $\sigma$. The approach used here is to define $\sigma$ as a sigmoid function,.
|
||||
To calculate this, we first define the kernel function as $\sigma '$, and the CDF of the kernel function as $\sigma$. The approach used here is to define $\sigma$ as a sigmoid function.
|
||||
|
||||
\begin{equation}
|
||||
\sigma(x ; v) = \left( 1 + \exp(-vx)\right)^{-1} \qquad \sigma'(x ; v) = \frac{v \cdot \exp(-v \cdot x)}{(1 + \exp(-vx))^2}
|
||||
@@ -358,9 +376,9 @@ We use bisection, which requires no derivative evaluation and is guaranteed to c
|
||||
for any strictly monotone CDF; the bracket construction and iteration are detailed in
|
||||
Appendix~\ref{appendix:kde-bisection}.
|
||||
|
||||
We can now relatively easily calculate $\hat{F}_1(\tau ; \mathbf{X}_1)$ using the formulat for $\hat{F}_{KDE}(\tau ; \mathbf{X})$ above.
|
||||
We can now trivially compute $\hat{F}_1(\tau ; \mathbf{X}_1)$ using the formula for $\hat{F}_{KDE}(\tau ; \mathbf{X})$ above.
|
||||
|
||||
Importantly, we must save the calculation of $\tau$ for the backward derivation, as well as the calculated bandwidths $v_1, v_0$.
|
||||
Note that $\tau$ is cached for use in the backward pass, along with the computed bandwidths $v_1, v_0$.
|
||||
|
||||
|
||||
\subsubsection{Gradient Derivation}
|
||||
@@ -408,7 +426,7 @@ A structural consequence of the ROLL formulation is that gradient mass is automa
|
||||
balanced between the two classes, regardless of their relative sizes in the batch.
|
||||
This stands in contrast to standard losses such as cross-entropy, where the total gradient
|
||||
contribution from each class is proportional to its count, making heavy class imbalance
|
||||
a direct source of biased updates. Therefor, instead of needing to compensate for this
|
||||
a direct source of biased updates. Therefore, instead of needing to compensate for this
|
||||
with loss-weighting per-class, fitting updates are made based equally off the results
|
||||
of both classes.
|
||||
|
||||
@@ -430,44 +448,7 @@ independently of $|\mathcal{B}_0|$ and $|\mathcal{B}_1|$.
|
||||
\end{proposition}
|
||||
|
||||
\begin{proof}
|
||||
We treat each class separately via a uniform-shift argument.
|
||||
|
||||
\medskip\noindent\textit{Sum over $\mathcal{B}_0$.}
|
||||
From \Cref{eq:roll-gradient}, for $y_i = 0$:
|
||||
\[
|
||||
\frac{\partial \mathcal{L}}{\partial f_\theta(\mathbf{x}_i)}
|
||||
= \frac{\partial \hat{F}_1(\tau)}{\partial \tau}
|
||||
\cdot \frac{\partial \tau}{\partial f_\theta(\mathbf{x}_i)}
|
||||
\]
|
||||
We claim $\sum_{\mathbf{x}_i \in \mathcal{B}_0}\frac{\partial \tau}{\partial f_\theta(\mathbf{x}_i)} = 1$.
|
||||
Shift all negative scores uniformly: $f_\theta(\mathbf{x}_i) \to f_\theta(\mathbf{x}_i) + \delta$ for
|
||||
$\mathbf{x}_i \in \mathcal{B}_0$. By translation invariance, $\hat{F}_0$ simply translates, so
|
||||
$\tau = \hat{F}_0^{-1}(1-\alpha)$ shifts by exactly $\delta$. Differentiating with respect to $\delta$ at
|
||||
$\delta = 0$:
|
||||
\[
|
||||
1 = \frac{\mathrm{d}\tau}{\mathrm{d}\delta}\bigg|_{\delta=0}
|
||||
= \sum_{\mathbf{x}_i \in \mathcal{B}_0} \frac{\partial \tau}{\partial f_\theta(\mathbf{x}_i)}
|
||||
\]
|
||||
Summing the $y_i = 0$ gradients therefore gives:
|
||||
\begin{equation}
|
||||
\sum_{\mathbf{x}_i \in \mathcal{B}_0}
|
||||
\frac{\partial \mathcal{L}}{\partial f_\theta(\mathbf{x}_i)}
|
||||
= \frac{\partial \hat{F}_1(\tau)}{\partial \tau} \cdot 1
|
||||
= \frac{\partial \hat{F}_1(\tau)}{\partial \tau}
|
||||
\label{eq:grad-balance-b0}
|
||||
\end{equation}
|
||||
|
||||
\medskip\noindent\textit{Sum over $\mathcal{B}_1$.}
|
||||
For $y_i = 1$, $\frac{\partial \mathcal{L}}{\partial f_\theta(\mathbf{x}_i)} = \frac{\partial \hat{F}_1(\tau)}{\partial f_\theta(\mathbf{x}_i)}$.
|
||||
Shifting all positive scores by $\delta$ and applying translation invariance of $\hat{F}_1$:
|
||||
$\hat{F}_1(\tau\,;\,\mathbf{s}+\delta\mathbf{1}) = \hat{F}_1(\tau-\delta\,;\,\mathbf{s})$.
|
||||
Differentiating at $\delta = 0$:
|
||||
\[
|
||||
\sum_{\mathbf{x}_i \in \mathcal{B}_1}
|
||||
\frac{\partial \hat{F}_1(\tau)}{\partial f_\theta(\mathbf{x}_i)}
|
||||
= -\frac{\partial \hat{F}_1(\tau)}{\partial \tau}
|
||||
\]
|
||||
Combining with \eqref{eq:grad-balance-b0} gives \eqref{eq:grad-balance}.
|
||||
See \Cref{appendix:grad-balance}.
|
||||
\end{proof}
|
||||
|
||||
The translation-invariance assumption holds for both the Gaussian distribution,
|
||||
@@ -479,7 +460,7 @@ is equivalent to an equal shift in $\tau$. The condition is necessary: without i
|
||||
score shift need not produce a rigid translation of $\hat{F}_k$, so the shift argument in the
|
||||
proof breaks down and the gradient sums need not be equal.
|
||||
|
||||
The Beta distribution only accepts values in the interval $\[0, 1\]$. Since it
|
||||
The Beta distribution only accepts values in the interval $[0, 1]$. Since it
|
||||
is mapped only over an interval, the translation-invariance assumption does not hold
|
||||
for this distribution.
|
||||
|
||||
@@ -625,5 +606,3 @@ distribution, producing very low gradient magnitudes and stalling convergence.
|
||||
|
||||
The solution was to use a bandwidth scheduler. Starting with large bandwidths resulted in gradients that spread out more evenly across all scores, and fewer non-convergence runs occurred. The bandwidth scheduler artificially scales up the bandwidth at the beginning to speed up initial training, and then tapers back to restore the intended bandwidth in order to maximise results.
|
||||
|
||||
|
||||
% TODO: algorithm box — full training loop (forward → ROLL loss → custom backward → weight update)
|
||||
|
||||
Reference in New Issue
Block a user