Fixes fixes fixes

content/appendixs/appendix_kde_grad.tex

@@ -140,3 +140,38 @@ entirely in terms of sigmoid values already cached from the forward pass:
 
 Substituting \eqref{eq:kde-grad-y1}, \eqref{eq:kde-dF1-dtau}, and \eqref{eq:kde-grad-tau}
 into \eqref{eq:roll-gradient} yields \eqref{eq:kde-grad-combined}.
+
+\paragraph{Numerical stability of the efficient form.}
+\label{sec:kde-grad-stability}
+
+\eqref{eq:kde-grad-tau-efficient} is vulnerable in floating point: $\sigma_0(\tau-f_i)$
+rounds to exactly $1$ once its argument $v_0(\tau-f_i)$ exceeds $\approx 17$ in single
+precision, so $1-\sigma_0(\tau-f_i)$ evaluates to exactly $0$.
+$\sigma_0'$ itself stays representable until the argument exceeds $\approx 88$, meaning we are giving
+up orders of magnitude of valid numerical representation. In extreme cases, such as those
+where a class is well-separated from the threshold $\tau$, and multiple scores fall
+over $5$ deviations away from said threshold, it is possible for many multiple scorewise
+gradients to fail, returning a value of $0$ instead of returning a low magnitude, yet usable, gradient.
+
+The ratio being computed is unchanged; only the arithmetic path is. Writing
+$z_i = v_0(\tau-f_\theta(\mathbf{x}_i))$ and using
+$\sigma'(z)=\sigma(z)(1-\sigma(z))$ together with the stable log-sigmoid identity
+$\log\sigma(z) = -\log(1+e^{-z})$:
+\begin{equation}
+  \log\sigma_0'(z_i) = -|z_i| - 2\log\bigl(1+e^{-|z_i|}\bigr) =: \ell_i
+  \label{eq:kde-grad-log-sigma-prime}
+\end{equation}
+(valid for either sign of $z_i$, since $\sigma'$ is even), which is safe to evaluate for
+any $|z_i|$: the only large-magnitude term is the linear $-|z_i|$, while
+$\log(1+e^{-|z_i|}) \in [0,\log 2]$ regardless of $|z_i|$. The normalised weight is then
+recovered by exponentiating exactly once, inside a $\operatorname{softmax}$-style
+normalisation that is itself overflow-safe:
+\begin{equation}
+  \frac{\sigma_0'(z_i)}{\sum_j \sigma_0'(z_j)} = \exp\Bigl(\ell_i - \operatorname*{logsumexp}_j \ell_j\Bigr).
+  \label{eq:kde-grad-softmax}
+\end{equation}
+This has no failure point short of the representable exponent range. The denominator
+can never collapse as it can under \eqref{eq:kde-grad-tau-efficient}.
+\eqref{eq:kde-grad-tau-efficient} is retained above as the clearest statement of the
+derivation; \eqref{eq:kde-grad-log-sigma-prime}, \eqref{eq:kde-grad-softmax} is what is
+implemented.