aner/thesis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fixes fixes fixes

Browse Source
This commit is contained in:
Aner Zakobar
2026-09-09 18:34:20 +03:00
parent 33d55169ef
commit f4de5a9e8a
5 changed files with 391 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/appendixs/appendix_kde_bisection.tex
+77 -8
View File
@@ -2,15 +2,51 @@ We describe the bisection procedure used to compute $\tau = \hat{F}_0^{-1}(1-\al
in the KDE-ROLL forward pass (\Cref{sec:roll-kde-forward}).
\paragraph{Bracket initialisation.}
Because $\sigma(u;\,v) \to 0$ exponentially as $u \to -\infty$ and $\sigma(u;\,v) \to 1$
as $u \to +\infty$, with tails decaying within roughly $10/v_0$ of the data range, the
interval
A fixed margin of $10/v_0 = 10h_0$ beyond the data range,
\[
\mathrm{lo} = \min(\mathbf{s}^{(0)}) - \tfrac{10}{v_0},
\mathrm{lo} = \min(\mathbf{s}^{(0)}) - 10h_0,
\qquad
\mathrm{hi} = \max(\mathbf{s}^{(0)}) + \tfrac{10}{v_0}
\mathrm{hi} = \max(\mathbf{s}^{(0)}) + 10h_0,
\]
is guaranteed to satisfy $\hat{F}_0(\mathrm{lo}) < 1-\alpha < \hat{F}_0(\mathrm{hi})$.
only guarantees $\hat{F}_0(\mathrm{lo}) < 1-\alpha < \hat{F}_0(\mathrm{hi})$ for
$1-\alpha \in (\sigma(-10;1),\,\sigma(10;1)) \approx (4.5\times10^{-5},\,1-4.5\times10^{-5})$,
since that is the range the sigmoid tails have decayed into at distance $10h_0$ — not for
every $\alpha \in (0,1)$ as a fixed margin might suggest.
Instead we use an exact, target-dependent bracket. For any $p \in (0,1)$, define
\[
\mathrm{lo}(p) = \min(\mathbf{s}^{(0)}) + h_0\log\!\frac{p}{1-p},
\qquad
\mathrm{hi}(p) = \max(\mathbf{s}^{(0)}) + h_0\log\!\frac{p}{1-p}.
\]
\begin{proposition}[Exact bracket containment]
\label{prop:bisection-bracket}
If $\min(\mathbf{s}^{(0)}) < \max(\mathbf{s}^{(0)})$, then for every $p \in (0,1)$,
\[
\hat{F}_0(\mathrm{lo}(p)) < p < \hat{F}_0(\mathrm{hi}(p)).
\]
\end{proposition}
\begin{proof}
$\sigma(u;v_0)$ is strictly increasing in $u$ (its derivative $v_0\,\sigma(1-\sigma) > 0$).
Direct substitution, using $v_0 h_0 = 1$, gives
$\sigma\bigl(\mathrm{lo}(p) - \min(\mathbf{s}^{(0)});\,v_0\bigr)
= \sigma\bigl(\log\tfrac{p}{1-p};\,1\bigr)
= \bigl(1+\tfrac{1-p}{p}\bigr)^{-1} = p$
exactly. For any other score $s_j > \min(\mathbf{s}^{(0)})$,
$\mathrm{lo}(p) - s_j < \mathrm{lo}(p) - \min(\mathbf{s}^{(0)})$, so monotonicity gives
$\sigma(\mathrm{lo}(p)-s_j;v_0) < p$. Averaging over $\mathcal{B}_0$, with at least one
score exceeding the minimum, $\hat{F}_0(\mathrm{lo}(p)) < p$ strictly. The symmetric
argument at $\max(\mathbf{s}^{(0)})$ gives $\hat{F}_0(\mathrm{hi}(p)) > p$.
\end{proof}
Setting $p = 1-\alpha$ gives a bracket valid for every $\alpha \in (0,1)$, unlike the
fixed-margin version above. The degenerate case $\min(\mathbf{s}^{(0)}) = \max(\mathbf{s}^{(0)})$
— a batch with constant negative-class scores — is excluded from
\Cref{prop:bisection-bracket} above; it also breaks the Silverman bandwidth estimate
itself ($h_0 = 0$), and is addressed as part of the degenerate-batch handling still
outstanding for \Cref{sec:kde-bandwidth}.
\paragraph{Bisection iterations.}
At each step we evaluate the midpoint,
@@ -24,7 +60,40 @@ and update the bracket:
\mathrm{hi} \leftarrow \tau_{\mathrm{mid}} & \text{otherwise.}
\end{cases}
\]
Iteration continues until $\lvert\hat{F}_0(\tau_{\mathrm{mid}};\,\mathcal{B}_0)-(1-\alpha)\rvert < \varepsilon$
(with $\varepsilon = 10^{-4}$), or until a maximum iteration count is reached.
\paragraph{Stopping criterion.}
A residual check on $\hat{F}_0$ alone, $\lvert\hat{F}_0(\tau_{\mathrm{mid}})-(1-\alpha)\rvert < \varepsilon$,
certifies that the \emph{threshold} $\tau$ has been located accurately, but not that the
\emph{loss} $\hat{F}_1(\tau)$ has: if $\hat{F}_0$ is comparatively flat near the target
(e.g.\ $v_0 \ll v_1$, or the negative-class kernel density is low at $\tau$), a wide range
of $\tau$ values can satisfy the residual bound while $\hat{F}_1(\tau)$ still varies
substantially across that range.
Instead, we exploit that $\hat{F}_1$ is itself a CDF — monotonically increasing — and
that the bracket invariant $\hat{F}_0(\mathrm{lo}) < 1-\alpha < \hat{F}_0(\mathrm{hi})$
keeps the true root $\tau^\star$ inside $[\mathrm{lo},\mathrm{hi}]$ throughout. Hence
\[
\hat{F}_1(\mathrm{lo}) \;\leq\; \hat{F}_1(\tau^\star) \;\leq\; \hat{F}_1(\mathrm{hi})
\]
regardless of exactly where $\tau^\star$ falls in the bracket. Iteration continues until
\begin{equation}
\hat{F}_1(\mathrm{hi};\,\mathcal{B}_1) - \hat{F}_1(\mathrm{lo};\,\mathcal{B}_1) < \varepsilon_L
\label{eq:bisection-loss-cert}
\end{equation}
(with $\varepsilon_L = 10^{-4}$), or until a maximum iteration count is reached. Unlike a
Lipschitz-based bound (e.g.\ using $\max_u \sigma'(u;v_1) = v_1/4$ to bound $\hat{F}_1$'s
slope and stopping purely on bracket width), \eqref{eq:bisection-loss-cert} is exact
rather than a worst-case estimate, at the cost of two extra $\hat{F}_1$ evaluations per
iteration.
\paragraph{Why the residual-only rule fails.} Take $\mathcal{B}_0 = \{-30,-25,5,45,60\}$
with $v_0 = 0.01$ (wide, smooth kernel) and $\alpha=\tfrac12$; the residual-only rule with
$\varepsilon=10^{-4}$ stops after 8 iterations at $\tau \approx 10.918$, versus the
converged root $\tau^\star \approx 10.932$, a gap of only $0.014$ in $\tau$. But with a
sharp positive-class kernel ($v_1 = 1000$) and a cluster of $\mathcal{B}_1$ sitting inside
that gap, $\hat{F}_1(\tau)$ differs by $0.40$ between the two: the reported loss is wrong
by nearly half its range, despite the threshold residual being $8\times$ tighter than
$\varepsilon$. Certifying via \eqref{eq:bisection-loss-cert} instead catches this
directly, converging to $\hat{F}_1$ accurate to $3\times 10^{-5}$.
The method converges unconditionally and requires no derivative evaluation, in contrast
to the Newton--Raphson alternative documented in Appendix~\ref{appendix:kde-nr}.
content/appendixs/appendix_kde_grad.tex
+35
View File
@@ -140,3 +140,38 @@ entirely in terms of sigmoid values already cached from the forward pass:
Substituting \eqref{eq:kde-grad-y1}, \eqref{eq:kde-dF1-dtau}, and \eqref{eq:kde-grad-tau}
into \eqref{eq:roll-gradient} yields \eqref{eq:kde-grad-combined}.
\paragraph{Numerical stability of the efficient form.}
\label{sec:kde-grad-stability}
\eqref{eq:kde-grad-tau-efficient} is vulnerable in floating point: $\sigma_0(\tau-f_i)$
rounds to exactly $1$ once its argument $v_0(\tau-f_i)$ exceeds $\approx 17$ in single
precision, so $1-\sigma_0(\tau-f_i)$ evaluates to exactly $0$.
$\sigma_0'$ itself stays representable until the argument exceeds $\approx 88$, meaning we are giving
up orders of magnitude of valid numerical representation. In extreme cases, such as those
where a class is well-separated from the threshold $\tau$, and multiple scores fall
over $5$ deviations away from said threshold, it is possible for many multiple scorewise
gradients to fail, returning a value of $0$ instead of returning a low magnitude, yet usable, gradient.
The ratio being computed is unchanged; only the arithmetic path is. Writing
$z_i = v_0(\tau-f_\theta(\mathbf{x}_i))$ and using
$\sigma'(z)=\sigma(z)(1-\sigma(z))$ together with the stable log-sigmoid identity
$\log\sigma(z) = -\log(1+e^{-z})$:
\begin{equation}
\log\sigma_0'(z_i) = -|z_i| - 2\log\bigl(1+e^{-|z_i|}\bigr) =: \ell_i
\label{eq:kde-grad-log-sigma-prime}
\end{equation}
(valid for either sign of $z_i$, since $\sigma'$ is even), which is safe to evaluate for
any $|z_i|$: the only large-magnitude term is the linear $-|z_i|$, while
$\log(1+e^{-|z_i|}) \in [0,\log 2]$ regardless of $|z_i|$. The normalised weight is then
recovered by exponentiating exactly once, inside a $\operatorname{softmax}$-style
normalisation that is itself overflow-safe:
\begin{equation}
\frac{\sigma_0'(z_i)}{\sum_j \sigma_0'(z_j)} = \exp\Bigl(\ell_i - \operatorname*{logsumexp}_j \ell_j\Bigr).
\label{eq:kde-grad-softmax}
\end{equation}
This has no failure point short of the representable exponent range. The denominator
can never collapse as it can under \eqref{eq:kde-grad-tau-efficient}.
\eqref{eq:kde-grad-tau-efficient} is retained above as the clearest statement of the
derivation; \eqref{eq:kde-grad-log-sigma-prime}, \eqref{eq:kde-grad-softmax} is what is
implemented.
content/appendixs/appendix_score_normalization.tex
+143
View File
@@ -0,0 +1,143 @@
This appendix proves \Cref{prop:normalization-invariance} (\Cref{sec:score-normalization-proof}),
shows numerically that ISJ does not satisfy its scale-equivariance hypothesis
(\Cref{sec:score-normalization-isj}), and derives the chain-rule term that a frozen
normalisation constant $\hat\sigma_\mathcal{B}$ omits, together with a concrete instance
where omitting it is necessary to preserve \Cref{obs:gradient-sign-correctness}
(\Cref{sec:score-normalization-missing-term}).
\section{Proof of Proposition~\ref{prop:normalization-invariance}}
\label{sec:score-normalization-proof}
Write $s$ for the raw batch scores, $c := \hat\sigma_\mathcal{B}(s) > 0$, and
$\tilde{s} = s/c$. Let $v_k^A := 1/\hat{h}(s^{(k)})$ (raw path) and
$v_k^B := 1/\hat{h}(\tilde{s}^{(k)})$ (normalised path), and let $\tau^A, \tau^B$ solve
$\hat{F}_0(\tau; s^{(0)}, v_0^A) = 1-\alpha$ and
$\hat{F}_0(\tau; \tilde{s}^{(0)}, v_0^B) = 1-\alpha$ respectively.
By hypothesis $\hat{h}(\tilde{s}^{(k)}) = \hat{h}(s^{(k)}/c) = \hat{h}(s^{(k)})/c$, so
\begin{equation}
v_k^B = c \cdot v_k^A.
\label{eq:sn-v-scale}
\end{equation}
Substituting $\tau = \tau^A/c$ into $\hat{F}_0(\tau; \tilde{s}^{(0)}, v_0^B)$:
\[
v_0^B\bigl(\tau^A/c - \tilde{s}_j^{(0)}\bigr)
= c v_0^A\Bigl(\frac{\tau^A - s_j^{(0)}}{c}\Bigr)
= v_0^A\bigl(\tau^A - s_j^{(0)}\bigr)
\]
for every $j$, so $\hat{F}_0(\tau^A/c; \tilde{s}^{(0)}, v_0^B) = \hat{F}_0(\tau^A; s^{(0)}, v_0^A) = 1-\alpha$
— i.e.\ $\tau^A/c$ solves the normalised-path equation, and by uniqueness of the
bisection root, $\tau^B = \tau^A/c$ exactly. The same substitution gives, for every $j$,
\begin{equation}
v_k^B\bigl(\tau^B - \tilde{s}_j^{(k)}\bigr) = v_k^A\bigl(\tau^A - s_j^{(k)}\bigr),
\label{eq:sn-argument-match}
\end{equation}
i.e.\ every sigmoid-kernel argument is identical between the two paths. Consequently
$\sigma(\cdot)$ evaluates identically in both paths (so the loss value matches:
$\hat{F}_1(\tau^B;\tilde{s}^{(1)},v_1^B) = \hat{F}_1(\tau^A;s^{(1)},v_1^A)$), while
$\sigma_k'(u;v) = v\,\sigma(u;v)(1-\sigma(u;v))$ picks up exactly a factor of
$v_k^B/v_k^A = c$ by \eqref{eq:sn-v-scale}:
\begin{equation}
\sigma_k'\bigl(\tau^B - \tilde{s}_j^{(k)}; v_k^B\bigr)
= c \cdot \sigma_k'\bigl(\tau^A - s_j^{(k)}; v_k^A\bigr).
\label{eq:sn-derivative-scale}
\end{equation}
\paragraph{Case $y_i=1$.} By \eqref{eq:kde-grad-y1} and the frozen-$c$ chain rule
$\partial/\partial s_i = (1/c)\,\partial/\partial\tilde{s}_i$:
\[
\frac{\partial L^B}{\partial s_i}
= \frac{1}{c}\left(-\frac{1}{|\mathcal{B}_1|}\sigma_1'\bigl(\tau^B-\tilde{s}_i;v_1^B\bigr)\right)
\overset{\eqref{eq:sn-derivative-scale}}{=}
\frac{1}{c}\left(-\frac{c}{|\mathcal{B}_1|}\sigma_1'\bigl(\tau^A-s_i;v_1^A\bigr)\right)
= \frac{\partial L^A}{\partial s_i}.
\]
\paragraph{Case $y_i=0$.} By \eqref{eq:kde-grad-combined}, the gradient is
$\frac{1}{c}\cdot D_1^B \cdot
\dfrac{\sigma_0'(\tau^B-\tilde{s}_i;v_0^B)}{\sum_j \sigma_0'(\tau^B-\tilde{s}_j;v_0^B)}$,
where $D_1^B := \frac{1}{|\mathcal{B}_1|}\sum_j\sigma_1'(\tau^B-\tilde{s}_j;v_1^B) = c\cdot D_1^A$
by \eqref{eq:sn-derivative-scale}. The ratio's numerator and denominator both pick up the
same factor $c$ (again by \eqref{eq:sn-derivative-scale}), so the ratio itself is
unchanged:
\[
\frac{\partial L^B}{\partial s_i}
= \frac{1}{c}\cdot c D_1^A \cdot
\frac{\sigma_0'(\tau^A-s_i;v_0^A)}{\sum_j\sigma_0'(\tau^A-s_j;v_0^A)}
= D_1^A \cdot \frac{\sigma_0'(\tau^A-s_i;v_0^A)}{\sum_j\sigma_0'(\tau^A-s_j;v_0^A)}
= \frac{\partial L^A}{\partial s_i}.
\]
In both cases the outer $1/c$ from the chain rule is exactly cancelled by the $c$
introduced into $v_k^B$, confirming \Cref{prop:normalization-invariance}. (Verified
numerically to float32 precision — $\max_i|\partial L^A/\partial s_i - \partial L^B/\partial s_i|
\approx 5.6\times10^{-9}$ on a random batch, using Silverman's rule for $\hat{h}$.)
\section{ISJ Does Not Satisfy the Hypothesis}
\label{sec:score-normalization-isj}
Silverman's rule satisfies $\hat{h}(a\cdot x)=a\cdot\hat{h}(x)$ exactly, since
$\hat{h}\propto\hat\sigma(x)$ and $\hat\sigma(ax)=a\hat\sigma(x)$. ISJ, computed via
\texttt{KDEpy}'s \texttt{improved\_sheather\_jones} (the primary estimator,
\Cref{sec:kde-bandwidth}), does not: rescaling a fixed sample by $a$ and re-running ISJ
gives $\hat{h}(ax)/a$ ratios that vary with $a$ rather than staying constant:
\begin{center}
\begin{tabular}{rrr}
\toprule
scale $a$ & $\hat{h}_{\text{Silverman}}(ax)/a$ & $\hat{h}_{\text{ISJ}}(ax)/a$ \\
\midrule
$10^{-6}$ & $0.353732$ & $0.001157$ \\
$10^{-3}$ & $0.353732$ & $0.001155$ \\
$10^{0}$ & $0.353732$ & $0.116810$ \\
$10^{3}$ & $0.353732$ & $0.217671$ \\
$10^{6}$ & $0.353732$ & $0.217671$ \\
\bottomrule
\end{tabular}
\end{center}
Consequently, for a batch where ISJ (rather than the Silverman fallback) is used,
neither the loss value nor the gradient is invariant to normalisation: a directly
measured example (random batch, $n{=}10$) gives loss $0.734$ on raw scores versus
$0.767$ on normalised scores, and a maximum per-sample gradient difference of $0.014$
— both far from the exact invariance \Cref{prop:normalization-invariance} guarantees
for Silverman. This is the genuine numerical-stability role of normalisation for the
method as actually used: not ``preventing overflow via an unchanged, merely-rescaled
argument'' (\Cref{prop:normalization-invariance} shows that mechanism is a no-op under
exact scale-equivariance), but pinning ISJ's input to the fixed scale regime — roughly
unit variance — it behaves consistently at, rather than leaving it to whatever scale the
model's raw outputs happen to have at a given point in training.
\section{The Missing Chain-Rule Term for a Differentiated $\hat\sigma_\mathcal{B}$}
\label{sec:score-normalization-missing-term}
\Cref{sec:roll-numerical-stability} treats $c=\hat\sigma_\mathcal{B}$ as frozen:
gradients with respect to normalised scores are multiplied by $1/c$ alone. Since $c$ is
itself a function of every score in the batch, $\tilde{s}_j = s_j/c(s_1,\ldots,s_n)$,
the fully differentiated chain rule is
\begin{equation}
\frac{\partial L}{\partial s_i}
= \sum_j \frac{\partial L}{\partial \tilde{s}_j}\frac{\partial \tilde{s}_j}{\partial s_i}
= \frac{1}{c}\frac{\partial L}{\partial \tilde{s}_i}
\;-\; \frac{1}{c^2}\frac{\partial c}{\partial s_i}\sum_j s_j\, q_j,
\qquad q_j := \frac{\partial L}{\partial \tilde{s}_j},
\label{eq:sn-missing-term}
\end{equation}
using $\partial \tilde{s}_j/\partial s_i = \delta_{ij}/c - (s_j/c^2)\,\partial c/\partial s_i$.
For the sample standard deviation used in \Cref{sec:roll-numerical-stability},
$\partial c/\partial s_i = (s_i-\bar{s})/((n-1)c)$. The frozen-$c$ convention sets
$\partial c/\partial s_i := 0$ by declaration, which is exactly why the implemented
gradient (\Cref{sec:roll-numerical-stability}) has no such term.
If instead $\hat\sigma_\mathcal{B}$ is differentiated through, the term in
\eqref{eq:sn-missing-term} does not generally vanish. Unlike the bandwidth case, there
is no exact-cancellation argument available here: the symmetry that would zero it out
via Euler's identity ($\sum_j \tilde{s}_j\, q_j = 0$, from $L$ being homogeneous of
degree $0$ in $\tilde{s}$) requires the bandwidth to be differentiated through the same
rescaling too — which \Cref{sec:roll-kde-backward} deliberately does not do. Numerically
(random batch, $n{=}10$, one $y_i{=}1$ sample): the frozen-$c$ gradient for that sample
is $-0.00181$ — correctly satisfying \Cref{obs:gradient-sign-correctness} ($\leq 0$
required) — while the fully-differentiated-$c$ gradient for the same sample is
$+0.00020$, violating it. Freezing $c$ is therefore adopted for the same reason as
freezing $v_0,v_1$ (Appendix~\ref{appendix:bandwidth-freeze}): the alternative is not
merely more complex, it can produce gradients of the wrong sign.
content/method/method.tex
+131 -57
View File
@@ -286,20 +286,32 @@ Direct integration is not tractable, but the integral is well-approximated by av
\mathcal{L}_{\text{ROLL-AOC}} = \frac{1}{99}\sum_{i=1}^{99}\mathcal{L}_{\text{ROLL-TPR@FPR}}\!\left(\frac{i}{100}\right)
\label{eq:roll-aoc}
\end{equation}
Since $\operatorname{TPR}(\alpha)$ is monotone in $\alpha$ along the ROC curve,
$g(\alpha) := 1-\operatorname{TPR}(\alpha)$ is monotone and bounded in $[0,1]$, so
\eqref{eq:roll-aoc} approximates $\int_0^1 g(\alpha)\,d\alpha$ to within
$1/99 \approx 0.01$ by a standard Darboux-sum argument, assuming each per-threshold
term is evaluated exactly. This bound covers only the quadrature step: it does not
extend to the gradient \eqref{eq:roll-aoc-gradient}, nor does it account for the gap
between the KDE-fitted rates and the raw empirical scores (\Cref{sec:roll-instantiations})
or any residual numerical error in the per-threshold bisection for $\tau_k$.
The gradient follows by linearity of the sum:
\begin{equation}
\frac{\partial \mathcal{L}_{\text{ROLL-AOC}}(f_\theta(\mathbf{X}))}
{\partial f_\theta(\mathbf{x}_i)} = \frac{1}{99}\sum_{i=1}^{99}\frac{\partial \mathcal{L}_{\text{ROLL-TPR@FPR}}\!\left(f_\theta(\mathbf{X}) ; \frac{i}{100}\right)}
{\partial f_\theta(\mathbf{x}_i)} = \frac{1}{99}\sum_{r=1}^{99}\frac{\partial \mathcal{L}_{\text{ROLL-TPR@FPR}}\!\left(f_\theta(\mathbf{X}) ; \frac{r}{100}\right)}
{\partial f_\theta(\mathbf{x}_i)}
\label{eq:roll-aoc-gradient}
\end{equation}
(the summation index $r$ ranges over the $99$ thresholds; $i$, fixed throughout the sum,
indexes the sample being differentiated — distinct roles that \eqref{eq:roll-aoc} does
not need to distinguish, since it has no outer per-sample index to collide with.)
The concrete per-threshold gradient for the KDE instantiation is derived in
\Cref{sec:roll-instantiations}; gradients for the parametric cases are in
\Cref{appendix:gauss-grad,appendix:beta-roll}.
\Cref{fig:roll-aoc-roc} gives a geometric view of this objective.
\begin{figure}[H]
\begin{figure}[H]
\centering
\includegraphics[width=0.62\textwidth]{content/method/figures/roll_aoc_roc.pdf}
\caption{Geometric interpretation of ROLL-AOC. The ROC curve (orange) maps
@@ -312,7 +324,8 @@ The concrete per-threshold gradient for the KDE instantiation is derived in
segments show the per-threshold loss contribution
$1 - \operatorname{TPR}(\alpha_i) = \mathcal{L}_{\text{ROLL-TPR@FPR}}(\alpha_i)$
at each point. Averaging these contributions (99 in practice, 10 shown
for clarity) gives the value of \eqref{eq:roll-aoc}.}
for clarity) approximates the continuous integral, giving the value of
\eqref{eq:roll-aoc}.}
\label{fig:roll-aoc-roc}
\end{figure}
@@ -335,15 +348,16 @@ Parametric families, however, impose distributional assumptions on the score out
no trained model can be guaranteed to satisfy. The primary instantiation in this thesis
is therefore non-parametric: Kernel Density Estimation (KDE) places no assumption on the
shape of the score distribution and adapts to whatever the model produces at each training
step. The cost is a numerical CDF inversion and a custom backward pass; the gain is a
faithful surrogate for the ROC curve regardless of how scores evolve during training.
step. The cost is a numerical CDF inversion and a custom backward pass. The gain is a
surrogate that adapts to the score distribution's shape as it evolves during training,
without committing to a parametric assumption that may not hold.
\subsection{KDE ROLL}
\label{sec:roll-kde}
KDE estimates the score distribution directly from the observed batch scores, placing no
assumption on their shape. It is the primary instantiation of ROLL used throughout this
thesis: its non-parametric flexibility makes it the most faithful surrogate for the ROC
thesis: its non-parametric flexibility makes it the most adaptable surrogate for the ROC
curve when score distributions are unknown, at the cost of a numerical CDF inversion and
a custom backward pass evaluated 99 times per training step under ROLL-AOC.
@@ -437,13 +451,11 @@ Appendix~\ref{appendix:bandwidth-freeze} shows by counterexample.
% guarantees ROLL provides and under what assumptions they hold.
% NOTE - these properties are auto-generated, consider suggestions.
A structural consequence of the ROLL formulation is that gradient mass is automatically
balanced between the two classes, regardless of their relative sizes in the batch.
This stands in contrast to standard losses such as cross-entropy, where the total gradient
contribution from each class is proportional to its count, making heavy class imbalance
a direct source of biased updates. Therefore, instead of needing to compensate for this
with loss-weighting per-class, fitting updates are made based equally off the results
of both classes.
A structural consequence of the ROLL formulation is that the total signed
gradient contribution from each class cancels exactly, regardless of their relative
sizes in the batch. This stands in contrast to standard losses such as cross-entropy,
where the total gradient contribution from each class is proportional to its count,
making heavy class imbalance a direct source of biased updates.
\begin{proposition}[Gradient balance]
\label{prop:gradient-balance}
@@ -471,9 +483,15 @@ as well as the KDE-based distribution, but not for the Beta distribution.
For Gaussian MLE, a uniform shift in scores moves $\mu_k$ by the same amount while
leaving $\sigma_k$ unchanged, which translates $\hat{F}_k$; for KDE, each kernel term is
evaluated at $\tau - f_\theta(\mathbf{x}_j)$, so a uniform shift in the $f_\theta(\mathbf{x}_j)$
is equivalent to an equal shift in $\tau$. The condition is necessary: without it, a uniform
score shift need not produce a rigid translation of $\hat{F}_k$, so the shift argument in the
proof breaks down and the gradient sums need not be equal.
is equivalent to an equal shift in $\tau$. The condition is sufficient, not necessary: it is
what the proof uses to guarantee a rigid translation of $\hat{F}_k$, but the balance identity
can hold without it. For example, $\hat{F}_k(t;\mathbf{s}) = \Phi(t - 2\bar{s})$ (with
$\bar{s}$ the class mean) does not satisfy $\hat{F}_k(t;\mathbf{s}+\delta) = \hat{F}_k(t-\delta;\mathbf{s})$
— shifting the scores by $\delta$ shifts $\hat{F}_k$ by $2\delta$, not $\delta$ — yet applying
this same construction to both classes still yields $\sum_{\mathcal{B}_1}\partial L/\partial f_i
= -\sum_{\mathcal{B}_0}\partial L/\partial f_i$ exactly, since the extra factor of $2$ enters
both classes' gradients identically and cancels in the same shift argument. Translation
invariance is a convenient sufficient condition, not the weakest one that would do.
The Beta distribution only accepts values in the interval $[0, 1]$. Since it
is mapped only over an interval, the translation-invariance assumption does not hold
@@ -486,6 +504,19 @@ average of 99 per-threshold objectives each satisfying \eqref{eq:grad-balance},
is preserved by linearity of summation.
\end{corollary}
The identity \eqref{eq:grad-balance} constrains only the signed sum per class. It
does not by itself say that each class contributes equal gradient mass,
$\sum_i |\partial\mathcal{L}/\partial f_\theta(\mathbf{x}_i)|$. Equal gradient-mass is satisfied
when every sample within a class shares the same gradient sign, so that the signed and
absolute sums are equal in magnitude. For KDE-ROLL,
this extra property does hold (\Cref{obs:gradient-sign-correctness} below), so the
signed identity does translate into balanced gradient mass for the method used
throughout this thesis. However, notably, this does not hold true for the Gaussian formulation.
Additionally, \Cref{prop:gradient-balance} is a statement about the per-sample loss
gradients $\partial\mathcal{L}/\partial f_\theta(\mathbf{x}_i)$ at the model's output. It
does not extend to the resulting parameter gradients $\partial\mathcal{L}/\partial\theta$,
as this depends on the model's structure.
\paragraph{Gradient sign correctness.}
\label{para:gradient-sign-correctness}
@@ -518,28 +549,42 @@ the fully adaptive gradient violates this sign guarantee.
\label{para:gradient-locality}
A second structural consequence of the ROLL-KDE formulation, when optimizing at a
single operating threshold, is that gradient mass is \emph{concentrated near that
threshold $\tau$}. Samples whose scores lie close to $\tau$ receive larger gradient
updates, while samples already far on the correct side contribute very little.
single operating threshold, is that gradient magnitude concentrates near $\tau$, but
the mechanism differs between the two classes, and so does how literally the claim
holds.
This follows directly from \Cref{eq:kde-grad-combined}. For the positive class ($y_i = 1$),
the gradient magnitude is
This follows from \Cref{eq:kde-grad-combined}. For the positive class ($y_i = 1$), the
gradient magnitude is an absolute, unnormalised function of the sample's own
distance to $\tau$:
\[
\left|\frac{\partial \mathcal{L}}{\partial f_\theta(\mathbf{x}_i)}\right|
= \frac{1}{|\mathcal{B}_1|}\,\sigma_1'(f_\theta(\mathbf{x}_i) - \tau),
\]
which is a bell-shaped kernel PDF that peaks when $f_\theta(\mathbf{x}_i) = \tau$ and
decays to zero as the score moves away from the threshold in either direction.
For the negative class ($y_i = 0$), the gradient is proportional to
$\sigma_0'(\tau - f_\theta(\mathbf{x}_i))$ (normalized over $\mathcal{B}_0$), which is
again bell-shaped and centred on $\tau$.
a bell-shaped kernel PDF that peaks when $f_\theta(\mathbf{x}_i) = \tau$ and decays to
exactly $0$ as $|f_\theta(\mathbf{x}_i)-\tau|\to\infty$, regardless of where any other
sample lies. For this class, "far from $\tau$ implies negligible gradient" holds without
qualification.
The effect is that each gradient step focuses the model's attention on the region that
directly determines the operating-point performance: samples that could plausibly be
re-ranked relative to the threshold at the current step. Samples classified with strong
confidence relative to the threshold, with scores far enough away from said threshold,
receive small gradient updates, since a small change to the score of these samples will not
bring any change to the placement of the threshold itself, or the performance at said threshold.
For the negative class ($y_i = 0$), the gradient is instead a \emph{normalised share} of
the class's total kernel mass, $\sigma_0'(\tau-f_\theta(\mathbf{x}_i)) \big/
\sum_j \sigma_0'(\tau-f_\theta(\mathbf{x}_j))$, and this ratio doesn't vanish just
because $f_\theta(\mathbf{x}_i)$ is far from $\tau$: what determines
sample $i$'s share of the gradient mass is its position relative to the rest of $\mathcal{B}_0$,
not its absolute distance to $\tau$. If the whole negative class drifts far from $\tau$
together, every individual $\sigma_0'$ value shrinks, but so does their sum, and the
ratio can converge to a nonzero constant no matter how far the class has drifted.
Concretely, for two negative samples at $\tau+M$ and $\tau+M+1$ (a fixed unit
separation), the closer sample's normalised weight converges, as $M\to\infty$, to
$1/(1+e^{-v_0})$, $\approx 0.731$ at $v_0=1$, not to $0$.
However, if a single score drifts far from $\tau$ while the rest of it's class scores,
stay close, its individual score will converge to zero.
The effect, for both classes, is that each gradient step focuses the model's attention
on the samples whose re-ranking relative to $\tau$ would most change the operating-point
performance. For the positive class this is an absolute statement about each sample's
own distance to $\tau$, and for the negative class, it is a statement about each sample's
distance relative to its classmates. Typically, but not necessarily, this correlates
with its absolute distance to $\tau$.
This stands in contrast to standard losses such as cross-entropy, where the per-sample
gradient is determined by the predicted probability, with no direct dependence on an
@@ -547,27 +592,36 @@ operating threshold.
\begin{observation}[Gradient locality, single-threshold]
\label{obs:gradient-locality}
For KDE-ROLL at a single operating threshold, the gradient magnitude of sample
$\mathbf{x}_i$ is proportional to the kernel PDF evaluated at the distance between
$f_\theta(\mathbf{x}_i)$ and $\tau$. Samples nearest to $\tau$ receive the largest
updates; samples far from $\tau$ in either direction contribute negligible gradient.
For KDE-ROLL at a single operating threshold: (a) a positive-class sample's gradient
magnitude is the kernel PDF evaluated at its own distance to $\tau$, and vanishes as
that distance grows, independent of the rest of the batch. (b) a negative-class
sample's gradient magnitude is its kernel-PDF value \emph{relative to the sum over
$\mathcal{B}_0$} — samples near $\tau$ receive a larger share than those far from it
\emph{within the same batch}, but this relative weighting does not by itself vanish
for an individual sample merely because the whole class is far from $\tau$.
\end{observation}
\Cref{fig:threshold-locality} illustrates this for a simulated two-class setting. The
\Cref{fig:threshold-locality} illustrates this for a simulated two-class setting in
which both classes are broadly spread around $\tau$ — the typical case, and the one in
which the negative class's relative weighting (\Cref{obs:gradient-locality}b) and the
positive class's absolute decay (\Cref{obs:gradient-locality}a) happen to agree. The
upper panel shows the score distributions and the operating threshold; the lower panel
shows the gradient magnitude of each sample. Both classes exhibit a clear bell-shaped
concentration of gradient mass around $\tau$, with smooth decay on either side.
shows the gradient magnitude of each sample, with a clear bell-shaped concentration
around $\tau$ for both classes in this regime.
\begin{figure}[H]
\centering
\includegraphics[width=0.82\textwidth]{content/method/figures/threshold_weighting.pdf}
\caption{Gradient locality in KDE-ROLL. \textit{Upper}: score distributions for the
negative (blue, $y=0$) and positive (orange, $y=1$) classes; dashed line marks the
operating threshold $\tau$ at FPR $= \alpha$. \textit{Lower}: gradient magnitude
\caption{Gradient locality in KDE-ROLL, for a batch where both classes are spread
broadly around $\tau$. \textit{Upper}: score distributions for the negative (blue,
$y=0$) and positive (orange, $y=1$) classes; dashed line marks the operating
threshold $\tau$ at FPR $= \alpha$. \textit{Lower}: gradient magnitude
$|\partial\mathcal{L}/\partial f_\theta(\mathbf{x})|$ for each sample. Both classes
show a bell-shaped concentration of gradient mass near $\tau$, with samples far from
the threshold contributing negligible updates. Smooth curves are the kernel PDFs scaled
for reference.}
show a bell-shaped concentration of gradient mass near $\tau$ here; for the negative
class this reflects relative weighting within $\mathcal{B}_0$
(\Cref{obs:gradient-locality}b), which need not track absolute distance to $\tau$ if
the whole class is displaced together. Smooth curves are the kernel PDFs scaled for
reference.}
\label{fig:threshold-locality}
\end{figure}
@@ -596,19 +650,39 @@ deviation:
\qquad
\hat\sigma_{\mathcal{B}} = \sqrt{\operatorname{Var}(f_\theta(\mathbf{x}_i) : (\mathbf{x}_i, y_i) \in \mathcal{B})}.
\]
This keeps the values given to the sigmoid kernel ($v(\tau - \tilde{s}_j)$)
near unit scale, preventing floating-point overflow and underflow in the exponential
evaluations.
This keeps the values given to the sigmoid kernel at a consistent scale. Why that
matters depends entirely on which bandwidth estimator is in play.
The scaling has no effect on the loss value itself or derived gradients.
Silverman's rule derives the bandwidth proportionally to the data standard deviation,
$h \propto \hat\sigma \cdot n^{-1/5}$, so dividing all scores by $\hat\sigma_\mathcal{B}$
causes the bandwidth to scale by the same factor.
The product $v \cdot (\tau - s_j)$ (the only quantity entering the sigmoid) is
therefore invariant under the rescaling.
In the backward pass, gradients with respect to the normalised scores are multiplied
by $1/\hat\sigma_\mathcal{B}$ to apply the chain rule, providing gradients mathematically
identical to those that would have been obtained without normalisation.
\begin{proposition}[Normalisation invariance under scale-equivariant bandwidth]
\label{prop:normalization-invariance}
Let $\hat{h}$ be any bandwidth estimator satisfying $\hat{h}(a\cdot x) = a\cdot\hat{h}(x)$
for all $a>0$ (e.g.\ Silverman's rule, $h \propto \hat\sigma \cdot n^{-1/5}$). Under the
frozen-bandwidth convention of \Cref{sec:roll-kde-backward} and a frozen normalisation
constant $\hat\sigma_\mathcal{B}$, the loss value and every per-sample gradient
$\partial \mathcal{L}_{\text{ROLL-TPR@FPR}}/\partial f_\theta(\mathbf{x}_i)$ computed from
the normalised scores $\tilde{s}$ are exactly equal to those computed directly from the
raw scores $s$.
\end{proposition}
\begin{proof}
See Appendix~\ref{appendix:score-normalization}.
\end{proof}
For Silverman's rule, normalisation is therefore a pure floating-point reparametrisation:
it changes neither the loss nor the gradient used for training, so it cannot be what
prevents overflow for that estimator — there is nothing left for it to change.
ISJ, the primary bandwidth estimator (\Cref{sec:kde-bandwidth}), does \emph{not} satisfy
the proposition's hypothesis (Appendix~\ref{appendix:score-normalization} measures its
scale-dependence directly). For ISJ, both the loss value and the gradient genuinely
depend on whether the scores are normalised, so normalisation is doing real work: it
pins ISJ's input to the one scale regime it behaves consistently in, rather than leaving
it to drift with the model's raw output scale over training.
As with the bandwidths $v_0,v_1$, the normalisation constant $\hat\sigma_\mathcal{B}$ is
treated as frozen (stop-gradient) in the backward pass: gradients with respect to the
normalised scores are simply multiplied by $1/\hat\sigma_\mathcal{B}$. Differentiating
fully through $\hat\sigma_\mathcal{B}$ instead introduces a further correction term that,
like the bandwidth correction of \Cref{sec:roll-kde-backward}, can violate
\Cref{obs:gradient-sign-correctness} (Appendix~\ref{appendix:score-normalization}).
\subsection{Bandwidth Estimation for KDE}
\label{sec:kde-bandwidth}
main.tex
+5
View File
@@ -311,6 +311,11 @@
\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח י}{\quad\qquadKDE ROLL: גרדיאנט רוחב פס אדפטיבי ומוסכמת ההקפאה}}{{\beginL\thepage\endL}}}
\input{content/appendixs/appendix_bandwidth_freeze}
\chapter{Score Normalisation: Invariance, ISJ, and the Frozen Chain Rule}
\label{appendix:score-normalization}
\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח יא}{\quad\qquadנרמול ציונים: שקילות, ISJ וכלל השרשרת הקפוא}}{{\beginL\thepage\endL}}}
\input{content/appendixs/appendix_score_normalization}
\end{appendices}
%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%