aner/thesis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Wrote stuff?

Browse Source
This commit is contained in:
Aner Zakobar
2026-08-29 17:53:06 +03:00
parent 1aef8e0f11
commit 24d6fac79c
9 changed files with 191 additions and 197 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/method/method.tex
+27 -173
View File
@@ -155,8 +155,9 @@ score vector in order to obtain a differentiable model of each class's score dis
\end{equation*}
%
Let $\hat{F}_1$ and $\hat{F}_0$ denote the CDFs of $P_{\hat{\phi}_1}$ and $P_{\hat{\phi}_0}$,
respectively. Specific choices of $\mathcal{F}$ (Gaussian, Beta, KDE) are discussed in
\Cref{sec:roll-instantiations}.
respectively. The KDE instantiation is developed in \Cref{sec:roll-instantiations};
the Gaussian and Beta instantiations, including full derivations, are in
\Cref{appendix:gauss-grad,appendix:beta-roll}.
Using this notation, at a threshold $\tau$:
\begin{align}
@@ -253,7 +254,8 @@ Unifying both cases, the general gradient applicable to all ROLL instantiations
\label{eq:roll-gradient}
\end{equation}
The concrete form of each partial derivative depends on the choice of distribution family
$\mathcal{F}$, and is derived for each instantiation in \Cref{sec:roll-instantiations}.
$\mathcal{F}$; the KDE case is derived in \Cref{sec:roll-instantiations}, and the
parametric cases are in \Cref{appendix:gauss-grad,appendix:beta-roll}.
\subsection{ROLL-AOC: AUC Optimization}
\label{sec:roll-aoc}
@@ -285,8 +287,9 @@ The gradient follows by linearity of the sum:
{\partial f_\theta(\mathbf{x}_i)}
\label{eq:roll-aoc-gradient}
\end{equation}
The concrete per-threshold gradient used in the sum is derived for each distribution
family in \Cref{sec:roll-instantiations}.
The concrete per-threshold gradient for the KDE instantiation is derived in
\Cref{sec:roll-instantiations}; gradients for the parametric cases are in
\Cref{appendix:gauss-grad,appendix:beta-roll}.
%------------------------------------------------
@@ -294,179 +297,30 @@ family in \Cref{sec:roll-instantiations}.
\section{Distribution Instantiations}
\label{sec:roll-instantiations}
% Each subsection below instantiates the general ROLL framework with a specific
% distributional assumption. Gaussian and Beta are included as parametric
% special cases; KDE is the primary contribution and receives the fullest treatment.
\subsection{Gaussian ROLL}
\label{sec:roll-gaussian}
% Assume positive- and negative-class scores are Gaussian distributed. The
% TPR/FPR integrals reduce to closed-form expressions in terms of the normal
% CDF, giving a simple and fast-to-compute instantiation of ROLL. Note the
% limitations: the Gaussian assumption can be violated in practice, motivating
% the more flexible instantiations below.
A simple method of implementing the ROLL loss is to approximate the distribution
of the score distributions as gaussian. This method is simple, as there exist
simple closed-form estimations for the parameters of gaussian distributions, and
they are differentiable.
While the distributions of scores cannot be gauranteed to follow any form, there are
some instances where the score distributions might be similar or exactly gaussian.
For example, when finding linear separability between gaussian distributions, the resulting
scores will be distributed normally, and this method will fit exactly.
In ROLL-AOC, the Gaussian instantiation evaluates \eqref{eq:roll-tpr-at-fpr-gaussian}
at each of the 99 FPR grid points and averages the results.
\subsubsection{Forward}
\label{sec:roll-gaussian-forward}
% Instantiate \eqref{eq:roll-tpr-at-fpr} under Gaussian score distributions.
% Each class's scores are modelled as $\mathcal{N}(\mu_k, \sigma_k^2)$, fitted
% by MLE. The CDF is the standard normal CDF $\Phi$, giving a closed-form loss.
Firstly, we must estimate the parameters to fit the gaussian distributions
to the true and false populations of the score output.
\begin{equation}
\mu_0 = \frac{1}{|\mathcal{B}_0|}\sum_{\mathbf{x}_j \in \mathcal{B}_0}{f_{\theta}(\mathbf{x}_j)}
\qquad
\mu_1 = \frac{1}{|\mathcal{B}_1|}\sum_{\mathbf{x}_j \in \mathcal{B}_1}{f_{\theta}(\mathbf{x}_j)}
\end{equation}
\begin{equation}
\sigma_0^2 = \frac{1}{|\mathcal{B}_0|}\sum_{\mathbf{x}_j \in \mathcal{B}_0}(f_\theta(\mathbf{x}_j) - \mu_0)^2
\qquad
\sigma_1^2 = \frac{1}{|\mathcal{B}_1|}\sum_{\mathbf{x}_j \in \mathcal{B}_1}(f_\theta(\mathbf{x}_j) - \mu_1)^2
\end{equation}
Once these parameters are estimated, we can then define our ROLL objective as required.
The CDF of the normal distribution with mean $\mu$ and standard deviation $\sigma$ is:
\[
F(x) = \frac{1}{2}\left[1 + \text{erf}\!\left(\frac{x - \mu}{\sigma\sqrt{2}}\right)\right]
\]
and its inverse (quantile function) is:
\[
F^{-1}(p) = \mu + \sigma\sqrt{2}\,\text{ierf}(2p - 1)
\]
where $\operatorname{erf}(x) = \frac{2}{\sqrt{\pi}}\int_0^x e^{-t^2}\,\mathrm{d}t$ is the Gauss
error function and $\operatorname{ierf}$ denotes its functional inverse, satisfying
$\operatorname{ierf}(\operatorname{erf}(x)) = x$. Neither has a closed form, but both are
available in standard numerical libraries (e.g.\ \texttt{torch.erf} and
\texttt{torch.erfinv}). Plugging into the general ROLL framework, the threshold
$\tau$ achieving FPR $= \alpha$ and the resulting loss are:
\begin{align}
\tau &= \mu_0 + \sigma_0\sqrt{2}\,\text{ierf}(1 - 2\alpha) \nonumber \\
\mathcal{L}_{\text{ROLL-TPR@FPR}}^{\text{GAUSSIAN}}(f_{\theta}(\mathcal{B}) ; \alpha)
&= \frac{1}{2}\left[1 + \text{erf}\!\left(\frac{\tau - \mu_1}{\sigma_1\sqrt{2}}\right)\right]
\label{eq:roll-tpr-at-fpr-gaussian}
\end{align}
\subsubsection{Gradient Derivation}
\label{sec:roll-gaussian-backward}
The full derivation is given in Appendix~\ref{appendix:gauss-grad}.
Letting $\tau = \hat{F}_0^{-1}(1-\alpha) = \mu_0 + \sigma_0\sqrt{2}\,\operatorname{ierf}(1-2\alpha)$, the gradient is:
\begin{equation}
\frac{\partial \mathcal{L}_{\text{ROLL-TPR@FPR}}^{\text{GAUSSIAN}}}{\partial f_\theta(\mathbf{x}_i)} =
\begin{cases}
\dfrac{-1}{|\mathcal{B}_1|\,\sigma_1\sqrt{2\pi}}
\exp\!\left(-\dfrac{(\tau-\mu_1)^2}{2\sigma_1^2}\right)
\!\left(1 + \dfrac{(\tau-\mu_1)(f_\theta(\mathbf{x}_i)-\mu_1)}{\sigma_1^2}\right)
& \text{if } y_i = 1 \\[18pt]
\dfrac{1}{|\mathcal{B}_0|\,\sigma_1\sqrt{2\pi}}
\exp\!\left(-\dfrac{(\tau-\mu_1)^2}{2\sigma_1^2}\right)
\!\left(1 + \dfrac{(\tau-\mu_0)(f_\theta(\mathbf{x}_i)-\mu_0)}{\sigma_0^2}\right)
& \text{if } y_i = 0
\end{cases}
\label{eq:gauss-grad-combined}
\end{equation}
\subsection{Beta ROLL}
\label{sec:roll-beta}
Beta ROLL applies when model scores are constrained to the open interval $(0,1)$,
as is the case when $f_\theta$ uses a sigmoid output activation. The Beta distribution
is supported on $(0,1)$ and its two shape parameters allow it to capture a wide range
of unimodal and skewed score distributions that a Gaussian cannot represent.
In ROLL-AOC, Beta ROLL evaluates \eqref{eq:roll-tpr-at-fpr-beta} at each of the 99
FPR grid points and averages the results, with gradients propagated via automatic
differentiation.
\subsubsection{Forward}
\label{sec:roll-beta-forward}
Given score vectors $\mathbf{s}^{(k)} \in (0,1)^{|\mathcal{B}_k|}$ for class $k$, let
$\mathbf{t}^{(k)} = \mathbf{1} - \mathbf{s}^{(k)}$ denote the complementary scores.
For any vector $\mathbf{v}$, write $\bar{v} = \frac{1}{|\mathbf{v}|}\sum_i v_i$.
We estimate the shape parameters using the closed-form log-moment estimator
of~\cite{chen2022beta}: define for each class $k$,
\begin{equation}
D_k = \overline{s^{(k)}\!\ln s^{(k)}} - \bar{s}^{(k)}\cdot\overline{\ln s^{(k)}}
+ \overline{t^{(k)}\!\ln t^{(k)}} - \bar{t}^{(k)}\cdot\overline{\ln t^{(k)}}
\label{eq:beta-denom}
\end{equation}
\begin{equation}
\hat{a}_k = \frac{\bar{s}^{(k)}}{D_k}, \qquad \hat{b}_k = \frac{\bar{t}^{(k)}}{D_k}
\label{eq:beta-params}
\end{equation}
These estimators follow from two simultaneous moment conditions:
$\mathbb{E}[X] = a/(a+b)$ and
$\operatorname{Cov}(X,\ln X) + \operatorname{Cov}(Y,\ln Y) = 1/(a+b)$,
where $Y=1-X\sim\operatorname{Beta}(b,a)$.
Replacing expectations by sample means and solving in closed form yields~\eqref{eq:beta-params}.
The CDF of $\operatorname{Beta}(\hat{a}_k,\hat{b}_k)$ at $x\in(0,1)$ is the
regularised incomplete beta function:
\begin{equation}
\hat{F}_k(x) = I_x(\hat{a}_k,\hat{b}_k)
= \frac{B(x;\,\hat{a}_k,\hat{b}_k)}{B(\hat{a}_k,\hat{b}_k)},
\qquad B(x;\,a,b) = \int_0^x u^{a-1}(1-u)^{b-1}\,\mathrm{d}u
\label{eq:beta-cdf}
\end{equation}
Substituting into the general ROLL framework \eqref{eq:roll-tpr-at-fpr}:
\begin{equation}
\mathcal{L}_{\text{ROLL-TPR@FPR}}^{\text{BETA}}(f_\theta(\mathcal{B});\,\alpha)
= I_{\hat\tau}(\hat{a}_1,\hat{b}_1),
\qquad
\hat\tau = I^{-1}_{1-\alpha}(\hat{a}_0,\hat{b}_0)
\label{eq:roll-tpr-at-fpr-beta}
\end{equation}
where $I^{-1}_{1-\alpha}(a,b)$ denotes the $(1-\alpha)$-quantile of $\operatorname{Beta}(a,b)$.
\subsubsection{Gradient Derivation}
\label{sec:roll-beta-backward}
Unlike the Gaussian case, we do not derive closed-form gradients for Beta ROLL.
The regularised incomplete beta function $I_x(a,b)$ and its inverse are smooth
functions of all their arguments and are available as differentiable operations in
PyTorch (\texttt{torch.special.betainc} and \texttt{torch.special.betaincinv}).
Gradients of \eqref{eq:roll-tpr-at-fpr-beta} with respect to $f_\theta(\mathbf{x}_i)$
are therefore propagated via automatic differentiation through the estimator
\eqref{eq:beta-params}, the CDF evaluation, and the quantile inversion.
The abstract ROLL framework (\Cref{sec:roll-formulation}) accommodates any distribution
family $\mathcal{F}$ whose CDF and quantile function are differentiable with respect to
the model scores. As part of this work, we instantiated the framework for two parametric
families: a Gaussian model, fitted by maximum likelihood, and a Beta model, fitted by a
closed-form log-moment estimator~\cite{chen2022beta}. Both admit fast parameter estimation
and differentiable CDFs; for the Gaussian case we derive analytic gradients in full, and
for the Beta case gradients propagate via automatic differentiation. Complete derivations
are given in \Cref{appendix:gauss-grad,appendix:beta-roll}.
Parametric families, however, impose distributional assumptions on the score output that
no trained model can be guaranteed to satisfy. The primary instantiation in this thesis
is therefore non-parametric: Kernel Density Estimation (KDE) places no assumption on the
shape of the score distribution and adapts to whatever the model produces at each training
step. The cost is a numerical CDF inversion and a custom backward pass; the gain is a
faithful surrogate for the ROC curve regardless of how scores evolve during training.
\subsection{KDE ROLL}
\label{sec:roll-kde}
% KDE background: see \Cref{sec:kde-background} in \Cref{chapter:related_work}.
% The primary instantiation: non-parametric, no assumption on score shape.
% Discuss the trade-off between flexibility and computational cost, and how
% the bandwidth (see \Cref{sec:kde-bandwidth}) controls bias-variance.
Unfortunately, the score output of trained models remains unpredictable. A solution for this is to use a Kernel Density Estimation (KDE) for probability
estimation. This benefits from being far more adaptable to real-world distributions of outputs of our trained models. The tradeoff of course is computational
complexity - computing the gradient derivation in such a case is non-trivial.
KDE ROLL is the primary instantiation used in ROLL-AOC: its non-parametric flexibility
makes it the most accurate estimator of the ROC curve when score distributions are
unknown, at the cost of evaluating the custom backward pass 99 times per training step.
KDE estimates the score distribution directly from the observed batch scores, placing no
assumption on their shape. It is the primary instantiation of ROLL used throughout this
thesis: its non-parametric flexibility makes it the most faithful surrogate for the ROC
curve when score distributions are unknown, at the cost of a numerical CDF inversion and
a custom backward pass evaluated 99 times per training step under ROLL-AOC.
\subsubsection{Forward}
\label{sec:roll-kde-forward}