aner/thesis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Added sections on symmetry, better equation mapping.

Browse Source
This commit is contained in:
Aner Zakobar
2026-07-02 23:24:19 +03:00
parent 10f1bd53f1
commit fa20fe0eb7
1 changed files with 76 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/method/method.tex
+76 -44
View File
@@ -242,10 +242,17 @@ Therefore, to minimise FPR:
\label{eq:roll-fpr-at-tpr} \label{eq:roll-fpr-at-tpr}
\end{equation} \end{equation}
TODO explain the trick where we can only use one version - base off code. Observe that \eqref{eq:roll-tpr-at-fpr} and \eqref{eq:roll-fpr-at-tpr} share the same
functional structure: each evaluates one empirical CDF at the quantile of the other. This
symmetry means \eqref{eq:roll-fpr-at-tpr} can be reduced to \eqref{eq:roll-tpr-at-fpr} by
negating the model scores and exchanging the class labels. Concretely, replacing
$f_\theta(\mathbf{x})$ with $-f_\theta(\mathbf{x})$ and $y$ with $1 - y$ swaps the roles
of the two populations, turning a TPR@FPR objective into a FPR@TPR one. Because the
two objectives are therefore interchangeable at the implementation level, all derivations
and implementation details below are given for \eqref{eq:roll-tpr-at-fpr} only;
\eqref{eq:roll-fpr-at-tpr} follows by the same transformation applied to the inputs.
We now derive the gradient of \eqref{eq:roll-tpr-at-fpr} with respect to $f_\theta(\mathbf{x}_i)$. We now derive the gradient of \eqref{eq:roll-tpr-at-fpr} with respect to $f_\theta(\mathbf{x}_i)$.
As both loss forms are symmetric in structure, it is sufficient to treat the TPR@FPR case.
If $y_i = 1$, then $(\mathbf{x}_i, y_i) \in \mathcal{B}_1$ and $f_\theta(\mathbf{x}_i)$ has no If $y_i = 1$, then $(\mathbf{x}_i, y_i) \in \mathcal{B}_1$ and $f_\theta(\mathbf{x}_i)$ has no
effect on $\hat{F}_0^{-1}(\alpha)$. Therefore: effect on $\hat{F}_0^{-1}(\alpha)$. Therefore:
@@ -470,7 +477,6 @@ As before, to calculate the loss, we calculate:
\[ \[
\mathcal{L}_{\text{ROLL-TPR@FPR}}^{\text{KDE}}(f_{\theta}(\mathcal{B}) ; \alpha) = \hat{F}_1(\hat{F}_0^{-1}(\alpha)) \mathcal{L}_{\text{ROLL-TPR@FPR}}^{\text{KDE}}(f_{\theta}(\mathcal{B}) ; \alpha) = \hat{F}_1(\hat{F}_0^{-1}(\alpha))
\label{eq:roll-tpr-at-fpr}
\] \]
To calculate this, we first define the kernel function as $\sigma '$, and the CDF of the kernel function as $\sigma$. The approach used here is to define $\sigma$ as a sigmoid function,. To calculate this, we first define the kernel function as $\sigma '$, and the CDF of the kernel function as $\sigma$. The approach used here is to define $\sigma$ as a sigmoid function,.
@@ -522,93 +528,119 @@ Importantly, we must save the calculation of $\tau$ for the backward derivation,
% Gradients flow through the kernel evaluations back to the model scores. % Gradients flow through the kernel evaluations back to the model scores.
% This is the section that connects to \Cref{sec:roll-backward} motivation. % This is the section that connects to \Cref{sec:roll-backward} motivation.
Firstly, for the true population, we calculate the derivatives. We derive the concrete form of the two terms from \Cref{eq:roll-gradient} for the KDE instantiation,
where $\tau = \hat{F}_0^{-1}(\alpha ; \mathcal{B}_0)$.
\paragraph{Case $y_i = 1$ (positive class).}
When $\mathbf{x}_i \in \mathcal{B}_1$, the score $f_\theta(\mathbf{x}_i)$ appears directly in $\hat{F}_1$
and has no effect on $\tau$ (which depends only on $\mathcal{B}_0$). Therefore:
\[ \[
\dfrac{\partial \hat{F}_1(\hat{F}_0^{-1}(\alpha))}{\partial f_{\theta}(\mathbf{x}_i)} = \dfrac{\partial \hat{F}_1(\hat{F}_0^{-1}(\alpha))}{\partial f_{\theta}(\mathbf{x}_i)} =
\frac{\partial \frac{1}{|\mathcal{B}_1|}\sum_{\mathbf{x}_j \in \mathcal{B}_1} (\sigma_1(f_{\theta}(\mathbf{x}_j) - \tau))}{\partial f_{\theta}(\mathbf{x}_i)} \frac{\partial \frac{1}{|\mathcal{B}_1|}\sum_{\mathbf{x}_j \in \mathcal{B}_1} \sigma_1(f_{\theta}(\mathbf{x}_j) - \tau)}{\partial f_{\theta}(\mathbf{x}_i)}
\] \]
\[ \[
= \frac{\partial \left(\sum_{\mathbf{x}_j \in \mathcal{B}_1 ; j \neq i} \frac{1}{|\mathcal{B}_1|}(\sigma_1(f_{\theta}(\mathbf{x}_j) - \tau)) + \frac{1}{|\mathcal{B}_1|}\sigma_1(f_{\theta}(\mathbf{x}_i) - \tau)\right)}{\partial f_{\theta}(\mathbf{x}_i)} = \frac{\partial \left(\sum_{\mathbf{x}_j \in \mathcal{B}_1 ; j \neq i} \frac{1}{|\mathcal{B}_1|}\sigma_1(f_{\theta}(\mathbf{x}_j) - \tau) + \frac{1}{|\mathcal{B}_1|}\sigma_1(f_{\theta}(\mathbf{x}_i) - \tau)\right)}{\partial f_{\theta}(\mathbf{x}_i)}
\] \]
\begin{equation}
\frac{\partial \hat{F}_1(\hat{F}_0^{-1}(\alpha))}{\partial f_{\theta}(\mathbf{x}_i)} = \frac{1}{|\mathcal{B}_1|}\sigma_1'(f_{\theta}(\mathbf{x}_i) - \tau)
\label{eq:kde-grad-y1}
\end{equation}
\paragraph{Case $y_i = 0$ (negative class).}
When $\mathbf{x}_i \in \mathcal{B}_0$, the score $f_\theta(\mathbf{x}_i)$ does not appear directly in
$\hat{F}_1$, but influences it through the threshold $\tau$. Per \Cref{eq:roll-gradient}, we compute
each factor of the chain rule separately.
\medskip\noindent\textit{Derivative of $\hat{F}_1$ with respect to $\tau$.}
\[ \[
= \frac{1}{|\mathcal{B}_1|}\sigma_1'(f_{\theta}(\mathbf{x}_i) - \tau) \frac{\partial \hat{F}_1(\hat{F}_0^{-1}(\alpha))}{\partial \hat{F}_0^{-1}(\alpha)} = \frac{\partial \frac{1}{|\mathcal{B}_1|}\sum_{\mathbf{x}_j \in \mathcal{B}_1} \sigma_1(f_{\theta}(\mathbf{x}_j) - \tau)}{\partial \tau}
\] \]
Now, for the false population. As stated in the roll framework equation above (TODO LINK) we must first compute the derivative w.r.t the threshold $\tau = \hat{F}_{0}^{-1}(\alpha)$: \begin{equation}
\[
\frac{\partial \hat{F}_1(\hat{F}_0^{-1}(\alpha))}{\partial \hat{F}_0^{-1}(\alpha)} = \frac{\partial \frac{1}{|\mathcal{B}_1|}\sum_{\mathbf{x}_j \in \mathcal{B}_1} (\sigma_1(f_{\theta}(\mathbf{x}_j) - \tau))}{\partial \tau}
\]
\[
= -\frac{1}{|\mathcal{B}_1|}\sum_{\mathbf{x}_j \in \mathcal{B}_1}\sigma_1'(f_{\theta}(\mathbf{x}_j) - \tau) = -\frac{1}{|\mathcal{B}_1|}\sum_{\mathbf{x}_j \in \mathcal{B}_1}\sigma_1'(f_{\theta}(\mathbf{x}_j) - \tau)
\] \label{eq:kde-dF1-dtau}
\end{equation}
At this point, we must calculate the derivative of the threshold w.r.t each score from the false population. \medskip\noindent\textit{Derivative of $\tau$ with respect to $f_\theta(\mathbf{x}_i)$.}
Unfortunately for us, this involves calculating the derivative of a calculation we computed numerically. This requires differentiating a quantity computed numerically --- $\tau = \hat{F}_0^{-1}(\alpha ; \mathcal{B}_0)$
is found via Newton's method rather than in closed form.
In order to derive this, we can utilize our earlier computation, along with the inverse derivative rule: We apply the inverse derivative rule:
\[ \[
f'(x) = \frac{1}{f^{-1}'(f(x))} f'(x) = \frac{1}{f^{-1}'(f(x))}
\] \]
This equation shows the relationship between a function's derivative and its inverse. It shows that we may use the derivative of the This allows us to use the derivative of the inverse function instead. We need the inverse of
inverse function. However, in order to do this, we must first define what is the inverse we are looking for. $\hat{F}_0^{-1}$ viewed as a function of $f_\theta(\mathbf{x}_i)$ --- that is, an expression for
$f_\theta(\mathbf{x}_i)$ in terms of $\tau$, $\alpha$, and $\mathcal{B}_0 \setminus \{\mathbf{x}_i\}$.
As a reminder we are trying to calculate $\frac{\partial \hat{F}_0^{-1}(\alpha ; \mathcal{B}_0)}{\partial f_\theta(\mathbf{x}_i)}$. That is, we have a function that takes $\alpha, \mathcal{B}_0$ as parameters and calculates Solving $|\mathcal{B}_0|\cdot\alpha = \sum_{\mathbf{x}_j \in \mathcal{B}_0}\sigma_0(\tau - f_\theta(\mathbf{x}_j))$
our threshold $\tau$. We are trying to calculate the derivative of this function w.r.t $\mathbf{x}_i$. for $f_\theta(\mathbf{x}_i)$ gives:
Therefore, we must invert the function in order to calculate $\mathbf{x}_i$ given $\alpha, \tau$ and all $\mathcal{B}_0 \backslash \{\mathbf{x}_i\}$.
This inverse equation is given by:
\[ \[
\tau - \sigma^{-1}\left( |\mathcal{B}_0|\cdot\alpha - \sum_{\mathbf{x}_j \in \mathcal{B}_0 ; j \neq i}\sigma(\tau - f_{\theta}(\mathbf{x}_j))\right) f_\theta(\mathbf{x}_i) = \tau - \sigma_0^{-1}\left( |\mathcal{B}_0|\cdot\alpha - \sum_{\mathbf{x}_j \in \mathcal{B}_0 ; j \neq i}\sigma_0(\tau - f_{\theta}(\mathbf{x}_j))\right)
\] \]
Applying the inverse derivative rule to this inverse: Applying the inverse derivative rule to this inverse:
\[ \[
\frac{\partial \hat{F}_0^{-1}(\alpha ; \mathcal{B}_0)}{\partial f_\theta(\mathbf{x}_i)} = \frac{1}{\frac{\partial \left(\tau - \sigma^{-1}\left( |\mathcal{B}_0|\cdot \alpha - \sum_{\mathbf{x}_j \in \mathcal{B}_0 ; j \neq i}\sigma(\tau - f_{\theta}(\mathbf{x}_j))\right)\right)}{\partial \tau}} \frac{\partial \hat{F}_0^{-1}(\alpha ; \mathcal{B}_0)}{\partial f_\theta(\mathbf{x}_i)} = \frac{1}{\frac{\partial \left(\tau - \sigma_0^{-1}\left( |\mathcal{B}_0|\cdot \alpha - \sum_{\mathbf{x}_j \in \mathcal{B}_0 ; j \neq i}\sigma_0(\tau - f_{\theta}(\mathbf{x}_j))\right)\right)}{\partial \tau}}
\] \]
Applying the chain rule: Applying the chain rule:
\[ \[
= \frac{1}{\sigma^{-1}'\left( |\mathcal{B}_0|\cdot \alpha - \sum_{\mathbf{x}_j \in \mathcal{B}_0 ; j \neq i}\sigma(\tau - f_{\theta}(\mathbf{x}_j))\right)\cdot\left( \sum_{\mathbf{x}_j \in \mathcal{B}_0 ; j \neq i}\sigma'(\tau - f_{\theta}(\mathbf{x}_j))\right) + 1} = \frac{1}{\sigma_0^{-1}{}'\left( |\mathcal{B}_0|\cdot \alpha - \sum_{\mathbf{x}_j \in \mathcal{B}_0 ; j \neq i}\sigma_0(\tau - f_{\theta}(\mathbf{x}_j))\right)\cdot\left( \sum_{\mathbf{x}_j \in \mathcal{B}_0 ; j \neq i}\sigma_0'(\tau - f_{\theta}(\mathbf{x}_j))\right) + 1}
\] \]
Remembering the calculation of $\alpha$ we get: Since $|\mathcal{B}_0|\cdot\alpha = \sum_{\mathbf{x}_j \in \mathcal{B}_0}\sigma_0(\tau - f_\theta(\mathbf{x}_j))$,
the argument of $\sigma_0^{-1}{}'$ simplifies:
\[ \[
\alpha = \sum_{\mathbf{x}_j \in \mathcal{B}_0}\sigma(\tau - f_{\theta}(\mathbf{x}_j)) \Rightarrow |\mathcal{B}_0|\cdot \alpha - \sum_{\mathbf{x}_j \in \mathcal{B}_0 ; j \neq i}\sigma_0(\tau - f_{\theta}(\mathbf{x}_j)) = \sigma_0(\tau - f_{\theta}(\mathbf{x}_i))
\] \]
Therefore:
\[ \[
|\mathcal{B}_0|\cdot \alpha - \sum_{\mathbf{x}_j \in \mathcal{B}_0 ; j \neq i}\sigma(\tau - f_{\theta}(\mathbf{x}_j)) = \sigma(\tau - f_{\theta}(\mathbf{x}_i)) \frac{\partial \hat{F}_0^{-1}(\alpha ; \mathcal{B}_0)}{\partial f_\theta(\mathbf{x}_i)} = \frac{1}{\sigma_0^{-1}{}'\!\left( \sigma_0(\tau - f_{\theta}(\mathbf{x}_i)) \right)\cdot\left( \sum_{\mathbf{x}_j \in \mathcal{B}_0 ; j \neq i}\sigma_0'(\tau - f_{\theta}(\mathbf{x}_j))\right) + 1 }
\] \]
Therefore, our derivative to calculate from before is equal to: Substituting $\sigma_0^{-1}{}'(\sigma_0(u)) = \frac{1}{\sigma_0'(u)}$ and multiplying numerator and denominator
by $\sigma_0'(\tau - f_{\theta}(\mathbf{x}_i))$:
\begin{equation}
\frac{\partial \hat{F}_0^{-1}(\alpha ; \mathcal{B}_0)}{\partial f_\theta(\mathbf{x}_i)} =
\frac{\sigma_0'(\tau - f_{\theta}(\mathbf{x}_i))}{\displaystyle\sum_{\mathbf{x}_j \in \mathcal{B}_0}\sigma_0'(\tau - f_{\theta}(\mathbf{x}_j))}
\label{eq:kde-grad-tau}
\end{equation}
\[ \paragraph{Combined gradient.}
\frac{\partial \hat{F}_0^{-1}(\alpha ; \mathcal{B}_0)}{\partial f_\theta(\mathbf{x}_i)} = \frac{1}{\sigma^{-1}'\left( \sigma(\tau - f_{\theta}(\mathbf{x}_i)) \right)\cdot\left( \sum_{\mathbf{x}_j \in \mathcal{B}_0 ; j \neq i}\sigma'(\tau - f_{\theta}(\mathbf{x}_j))\right) + 1 }
\]
Substituting $\sigma^{-1}'(\sigma(u)) = \frac{1}{\sigma'(u)}$ and multiplying numerator and denominator by $\sigma'(\tau - f_{\theta}(\mathbf{x}_i))$: Substituting \Cref{eq:kde-grad-y1}, \Cref{eq:kde-dF1-dtau}, and \Cref{eq:kde-grad-tau}
into \Cref{eq:roll-gradient}:
\[ \begin{equation}
\frac{\partial \hat{F}_0^{-1}(\alpha ; \mathcal{B}_0)}{\partial f_\theta(\mathbf{x}_i)} = \frac{\sigma'(\tau - f_{\theta}(\mathbf{x}_i))}{\displaystyle\sum_{\mathbf{x}_j \in \mathcal{B}_0}\sigma'(\tau - f_{\theta}(\mathbf{x}_j))} \frac{\partial \mathcal{L}_{\text{ROLL-TPR@FPR}}}{\partial f_\theta(\mathbf{x}_i)} =
\] \begin{cases}
\dfrac{1}{|\mathcal{B}_1|}\,\sigma_1'(f_{\theta}(\mathbf{x}_i) - \tau)
& \text{if } y_i = 1 \\[14pt]
-\dfrac{1}{|\mathcal{B}_1|}\!\displaystyle\sum_{\mathbf{x}_j \in \mathcal{B}_1}\!\sigma_1'(f_{\theta}(\mathbf{x}_j) - \tau)
\;\cdot\;
\dfrac{\sigma_0'(\tau - f_{\theta}(\mathbf{x}_i))}{\displaystyle\sum_{\mathbf{x}_j \in \mathcal{B}_0}\sigma_0'(\tau - f_{\theta}(\mathbf{x}_j))}
& \text{if } y_i = 0
\end{cases}
\label{eq:kde-grad-combined}
\end{equation}
Which we can plug back into the ROLL derivation framework. where $\tau = \hat{F}_0^{-1}(\alpha ; \mathcal{B}_0)$.
%------------------------------------------------ %------------------------------------------------