aner/thesis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Hopefully final draft

Browse Source
This commit is contained in:
Aner Zakobar
2026-09-10 02:46:31 +03:00
parent f4de5a9e8a
commit 1d1b783090
10 changed files with 414 additions and 123 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/appendixs/appendix_score_normalization.tex
+12 -14
View File
@@ -26,7 +26,7 @@ Substituting $\tau = \tau^A/c$ into $\hat{F}_0(\tau; \tilde{s}^{(0)}, v_0^B)$:
= v_0^A\bigl(\tau^A - s_j^{(0)}\bigr)
\]
for every $j$, so $\hat{F}_0(\tau^A/c; \tilde{s}^{(0)}, v_0^B) = \hat{F}_0(\tau^A; s^{(0)}, v_0^A) = 1-\alpha$
— i.e.\ $\tau^A/c$ solves the normalised-path equation, and by uniqueness of the
i.e.\ $\tau^A/c$ solves the normalised-path equation, and by uniqueness of the
bisection root, $\tau^B = \tau^A/c$ exactly. The same substitution gives, for every $j$,
\begin{equation}
v_k^B\bigl(\tau^B - \tilde{s}_j^{(k)}\bigr) = v_k^A\bigl(\tau^A - s_j^{(k)}\bigr),
@@ -70,7 +70,7 @@ unchanged:
In both cases the outer $1/c$ from the chain rule is exactly cancelled by the $c$
introduced into $v_k^B$, confirming \Cref{prop:normalization-invariance}. (Verified
numerically to float32 precision — $\max_i|\partial L^A/\partial s_i - \partial L^B/\partial s_i|
numerically to float32 precision: $\max_i|\partial L^A/\partial s_i - \partial L^B/\partial s_i|
\approx 5.6\times10^{-9}$ on a random batch, using Silverman's rule for $\hat{h}$.)
\section{ISJ Does Not Satisfy the Hypothesis}
@@ -99,13 +99,13 @@ $10^{6}$ & $0.353732$ & $0.217671$ \\
Consequently, for a batch where ISJ (rather than the Silverman fallback) is used,
neither the loss value nor the gradient is invariant to normalisation: a directly
measured example (random batch, $n{=}10$) gives loss $0.734$ on raw scores versus
$0.767$ on normalised scores, and a maximum per-sample gradient difference of $0.014$
— both far from the exact invariance \Cref{prop:normalization-invariance} guarantees
$0.767$ on normalised scores, and a maximum per-sample gradient difference of $0.014$,
both far from the exact invariance \Cref{prop:normalization-invariance} guarantees
for Silverman. This is the genuine numerical-stability role of normalisation for the
method as actually used: not ``preventing overflow via an unchanged, merely-rescaled
argument'' (\Cref{prop:normalization-invariance} shows that mechanism is a no-op under
exact scale-equivariance), but pinning ISJ's input to the fixed scale regime — roughly
unit variance — it behaves consistently at, rather than leaving it to whatever scale the
exact scale-equivariance), but pinning ISJ's input to the fixed scale regime (roughly
unit variance) it behaves consistently at, rather than leaving it to whatever scale the
model's raw outputs happen to have at a given point in training.
\section{The Missing Chain-Rule Term for a Differentiated $\hat\sigma_\mathcal{B}$}
@@ -130,14 +130,12 @@ $\partial c/\partial s_i := 0$ by declaration, which is exactly why the implemen
gradient (\Cref{sec:roll-numerical-stability}) has no such term.
If instead $\hat\sigma_\mathcal{B}$ is differentiated through, the term in
\eqref{eq:sn-missing-term} does not generally vanish. Unlike the bandwidth case, there
is no exact-cancellation argument available here: the symmetry that would zero it out
via Euler's identity ($\sum_j \tilde{s}_j\, q_j = 0$, from $L$ being homogeneous of
degree $0$ in $\tilde{s}$) requires the bandwidth to be differentiated through the same
rescaling too — which \Cref{sec:roll-kde-backward} deliberately does not do. Numerically
(random batch, $n{=}10$, one $y_i{=}1$ sample): the frozen-$c$ gradient for that sample
is $-0.00181$ — correctly satisfying \Cref{obs:gradient-sign-correctness} ($\leq 0$
required) — while the fully-differentiated-$c$ gradient for the same sample is
\eqref{eq:sn-missing-term} does not generally vanish. There is no gaurantee
that this term does not overwhelm the first, flipping the sign and rendering the loss
useless. Numerically (random batch, $n{=}10$, one $y_i{=}1$ sample):
the frozen-$c$ gradient for that sample
is $-0.00181$, correctly satisfying \Cref{obs:gradient-sign-correctness} ($\leq 0$
required), while the fully-differentiated-$c$ gradient for the same sample is
$+0.00020$, violating it. Freezing $c$ is therefore adopted for the same reason as
freezing $v_0,v_1$ (Appendix~\ref{appendix:bandwidth-freeze}): the alternative is not
merely more complex, it can produce gradients of the wrong sign.