aner/thesis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Hopefully final draft

Browse Source
This commit is contained in:
Aner Zakobar
2026-09-10 02:46:31 +03:00
parent f4de5a9e8a
commit 1d1b783090
10 changed files with 414 additions and 123 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/appendixs/appendix_bandwidth_freeze.tex
+1 -1
View File
@@ -91,7 +91,7 @@ independent of the bandwidth.
Silverman's rule on $\mathcal{B}_1 = \{0,b\}$ ($|\mathcal{B}_1|=2$,
$\hat\sigma_1 = b/2$) gives $h_1 = \kappa b$ with $\kappa := 1.06 \cdot 2^{-1/5}/2$, i.e.\
$v_1 = 1/(\kappa b)$ — the bandwidth scales linearly with $b$.
$v_1 = 1/(\kappa b)$: the bandwidth scales linearly with $b$.
\paragraph{True adaptive loss.} Substituting $v_1 = 1/(\kappa b)$,
\[
content/appendixs/appendix_gauss_grad.tex
+31 -1
View File
@@ -1,6 +1,11 @@
We instantiate the general ROLL framework (\Cref{sec:roll-formulation}) with
Gaussian-distributed scores. Each class's scores are modelled as
$\mathcal{N}(\mu_k, \sigma_k^2)$, fitted by maximum-likelihood estimation.
$\mathcal{N}(\mu_k, \sigma_k^2)$, fitted by maximum-likelihood
estimation.\footnote{If a class's scores are literally constant, $\sigma_k=0$ and the
CDF and gradient expressions throughout this appendix are undefined (division by
$\sigma_k$ or $\sigma_k^2$). This edge case is unsupported in the current
implementation; it does not arise with real, continuously-valued model outputs, but
supporting it in general would require an explicit floor on $\sigma_k$.}
\paragraph{Forward pass.}
@@ -27,6 +32,31 @@ Substituting into the general ROLL objective \eqref{eq:roll-tpr-at-fpr}:
\label{eq:roll-tpr-at-fpr-gaussian}
\end{align}
\paragraph{Closed-form AOC.}
Unlike the general (KDE) case, the full AOC integral admits a closed form for the
Gaussian instantiation, bypassing the 99-point quadrature of \Cref{sec:roll-aoc}
entirely. Writing $\Phi$ for the standard normal CDF
($\Phi(x) = \tfrac{1}{2}[1+\operatorname{erf}(x/\sqrt{2})]$) and $\varphi$ for its
density, substitute \eqref{eq:roll-tpr-at-fpr-gaussian} into
$\mathcal{L}_{\text{AOC}} = \int_0^1 \mathcal{L}_{\text{ROLL-TPR@FPR}}(\alpha)\,d\alpha$
and change variables via $u = \Phi^{-1}(1-\alpha)$, so $\tau = \mu_0 + \sigma_0 u$:
\[
\mathcal{L}_{\text{AOC}}^{\text{Gaussian}}
= \int_{-\infty}^{\infty} \Phi\!\left(\frac{\mu_0-\mu_1+\sigma_0 u}{\sigma_1}\right)\varphi(u)\,du.
\]
Applying the standard identity
$\int_{-\infty}^{\infty}\Phi(a+bu)\,\varphi(u)\,du = \Phi\!\bigl(a/\sqrt{1+b^2}\bigr)$
with $a = (\mu_0-\mu_1)/\sigma_1$, $b = \sigma_0/\sigma_1$:
\begin{equation}
\mathcal{L}_{\text{AOC}}^{\text{Gaussian}} = \Phi\!\left(\frac{\mu_0-\mu_1}{\sqrt{\sigma_0^2+\sigma_1^2}}\right).
\label{eq:gauss-aoc-closed-form}
\end{equation}
(Verified numerically against direct quadrature of the integral.) We nonetheless use
the same 99-point average as every other instantiation in this thesis, for
implementation uniformity across distribution families rather than necessity
(\Cref{sec:roll-aoc}).
\paragraph{Gradient derivation.}
We derive the gradient of \eqref{eq:roll-tpr-at-fpr-gaussian} with respect to
content/appendixs/appendix_kde_bisection.tex
+5 -4
View File
@@ -10,7 +10,8 @@ A fixed margin of $10/v_0 = 10h_0$ beyond the data range,
\]
only guarantees $\hat{F}_0(\mathrm{lo}) < 1-\alpha < \hat{F}_0(\mathrm{hi})$ for
$1-\alpha \in (\sigma(-10;1),\,\sigma(10;1)) \approx (4.5\times10^{-5},\,1-4.5\times10^{-5})$,
since that is the range the sigmoid tails have decayed into at distance $10h_0$ — not for
since that is the range the sigmoid tails have decayed into at distance $10h_0$. This
does not hold true for
every $\alpha \in (0,1)$ as a fixed margin might suggest.
Instead we use an exact, target-dependent bracket. For any $p \in (0,1)$, define
@@ -42,8 +43,8 @@ argument at $\max(\mathbf{s}^{(0)})$ gives $\hat{F}_0(\mathrm{hi}(p)) > p$.
\end{proof}
Setting $p = 1-\alpha$ gives a bracket valid for every $\alpha \in (0,1)$, unlike the
fixed-margin version above. The degenerate case $\min(\mathbf{s}^{(0)}) = \max(\mathbf{s}^{(0)})$
— a batch with constant negative-class scores — is excluded from
fixed-margin version above. The degenerate case $\min(\mathbf{s}^{(0)}) = \max(\mathbf{s}^{(0)})$.
This represents a batch with constant negative-class scores. I is excluded from
\Cref{prop:bisection-bracket} above; it also breaks the Silverman bandwidth estimate
itself ($h_0 = 0$), and is addressed as part of the degenerate-batch handling still
outstanding for \Cref{sec:kde-bandwidth}.
@@ -68,7 +69,7 @@ certifies that the \emph{threshold} $\tau$ has been located accurately, but not
of $\tau$ values can satisfy the residual bound while $\hat{F}_1(\tau)$ still varies
substantially across that range.
Instead, we exploit that $\hat{F}_1$ is itself a CDF — monotonically increasing — and
Instead, we exploit that $\hat{F}_1$ is itself a CDF (monotonically increasing) and
that the bracket invariant $\hat{F}_0(\mathrm{lo}) < 1-\alpha < \hat{F}_0(\mathrm{hi})$
keeps the true root $\tau^\star$ inside $[\mathrm{lo},\mathrm{hi}]$ throughout. Hence
\[
content/appendixs/appendix_np_theory.tex
+118
View File
@@ -0,0 +1,118 @@
This appendix gives a Neyman-Pearson-theoretic account of why order-dependent losses (objectives that depend on the ranking of scores rather than on comparing each score
to a fixed threshold) avoid a real limitation of class-weighted
per-sample losses. \Cref{sec:np-class-weighting} shows that a weighted per-sample loss
commits to a Neyman-Pearson threshold $\eta$ chosen indirectly through the class
weights.
\Cref{sec:np-noise-invariance} shows that under label noise this implicit $\eta$ is
further, unpredictably distorted. Neither limitation is specific to \textsc{Roll}: they
apply to any per-sample-weighted loss, and the way around them (targeting an operating
point or optimizing AUC directly, without ever forming a class weight or comparing to a
fixed $\eta$) applies equally to \textsc{Roll}'s TPR@FPR and ROLL-AOC objectives and to
other AUC-optimizing methods such as \textsc{LibAUC}~\cite{yuan2021auc}.
\section{Class Weighting as an Implicit Choice of $\eta$}
\label{sec:np-class-weighting}
Recall from \Cref{sec:neyman-pearson} that the Neyman-Pearson lemma characterizes the
optimal test at a fixed false-positive rate as thresholding the likelihood ratio
$p(\mathbf{x}\mid y{=}1)/p(\mathbf{x}\mid y{=}0)$ against some constant $\eta$: a single
threshold, fixed once for the whole test, the same for every $\mathbf{x}$. We will not
need $\eta$'s exact value below, only that it exists, that it is one constant shared by
every sample, and that a weighted per-sample loss determines it indirectly through the
class weights.
\Cref{sec:imbalanced-tpr-fpr} noted that per-sample losses typically correct for class
imbalance by multiplying the loss on one class by a weight. This practice has a direct
reading in Neyman-Pearson terms: for a weighted per-sample loss, minimizing the pointwise
loss and thresholding the result is equivalent to thresholding the posterior
odds,\footnote{For weighted cross-entropy with weights $w_1,w_0$, minimizing
$w_1 P(y{=}1\mid\mathbf{x})(-\log q) + w_0 P(y{=}0\mid\mathbf{x})(-\log(1-q))$ over the
predicted probability $q$ gives $q^\star = w_1 P(y{=}1\mid\mathbf{x}) \big/
\bigl[w_1 P(y{=}1\mid\mathbf{x}) + w_0 P(y{=}0\mid\mathbf{x})\bigr]$. Thresholding
$q^\star > \tfrac12$ is equivalent to $w_1 P(y{=}1\mid\mathbf{x}) > w_0 P(y{=}0\mid\mathbf{x})$,
i.e. to the threshold on $O(\mathbf{x})$ stated below.} $O(\mathbf{x}) = P(y{=}1\mid\mathbf{x})/P(y{=}0\mid\mathbf{x})$,
against a constant set by the weight ratio:
\[
O(\mathbf{x}) > \eta, \qquad \eta = \frac{w_0}{w_1}.
\]
This exact value of $\eta$ follows from this particular (weighted cross-entropy) loss;
other weighted per-sample losses would fix it slightly differently. What matters below is
not the exact value but the qualitative fact common to all of them: $\eta$ is one constant,
the same for every sample, that moves with the chosen weight ratio. Choosing per-class
weights is therefore, knowingly or not, equivalent to choosing $\eta$ indirectly, and
without prior knowledge of the actual separability of the data.
Concretely, for any point $\mathbf{x}$, this means the classifier ends up treating
$\mathbf{x}$ as positive or negative purely according to which side of $\eta$ its own
posterior odds $O(\mathbf{x})$ falls on: samples with $O(\mathbf{x}) > \eta$ are pushed
toward the positive class, samples with $O(\mathbf{x}) < \eta$ toward the negative class,
regardless of how separable $\mathbf{x}$ actually is from the other class. Two samples with
nearly identical posterior odds straddling $\eta$ can therefore end up on opposite sides of
the decision rule purely because of where $\eta$ happens to land, and since $\eta$ is
fixed entirely by the chosen weight ratio, it is the weight choice, not any property of the
data, that decides the fate of every such borderline sample. This is the central difference
from an order-dependent loss: a per-sample loss must commit to some $\eta$ before training
even begins, and every sample's outcome is filtered through that fixed, weight-determined
cutoff.
Ordering-based objectives, such as our ROLL instantiations, avoid this altogether: rather
than fixing $\eta$ via class weights, they optimize for AUC or a target operating point
directly, and whatever $\eta$ this implies is discovered from the fitted score
distributions during training. No class weight, and no assumption linking class counts to
$\eta$, is needed. The same is true of any objective that optimizes AUC or a fixed
operating point directly, including \textsc{LibAUC}: none of these ever form a
class-weight ratio or compare a probability to a fixed $\eta$ in the first place.
\section{Class Weighting Under Label Noise: An Implicit, Distorted $\eta$}
\label{sec:np-noise-invariance}
\Cref{sec:np-class-weighting} showed that a weighted per-sample loss trained on clean
labels implicitly targets a threshold $\eta$ on the true posterior odds
$O(\mathbf{x}) = P(y{=}1\mid\mathbf{x})/P(y{=}0\mid\mathbf{x})$, set by the weight ratio.
In practice, however, training data is rarely clean. This section shows that under label
noise, the threshold actually reached is not the one chosen, regardless of how $\eta$
itself is defined.
Suppose labels are corrupted by class-conditional noise, the same model used in
\Cref{sec:noisy-labels} and in this thesis's noise experiments: each label is flipped with
probability $\rho \in [0,\tfrac12)$ depending only on its true class, independent of
$\mathbf{x}$. A model trained on the resulting noisy labels $\tilde y$ estimates the noisy
posterior $P(\tilde y{=}1\mid\mathbf{x})$, not the true one, so a weight ratio $w = w_0/w_1$
now thresholds $P(\tilde y{=}1\mid\mathbf{x})/P(\tilde y{=}0\mid\mathbf{x})$, not
$O(\mathbf{x})$. These two quantities are related by a fixed function of $\rho$
alone,\footnote{Since the noise depends only on $y$, $P(\tilde y{=}1\mid\mathbf{x}) =
(1-\rho)P(y{=}1\mid\mathbf{x}) + \rho P(y{=}0\mid\mathbf{x})$, and similarly
$P(\tilde y{=}0\mid\mathbf{x}) = \rho P(y{=}1\mid\mathbf{x}) + (1-\rho)P(y{=}0\mid\mathbf{x})$.
Dividing both by $P(y{=}0\mid\mathbf{x})$ and taking the ratio gives
$P(\tilde y{=}1\mid\mathbf{x})/P(\tilde y{=}0\mid\mathbf{x}) = h(O(\mathbf{x}))$ with
$h(O) = [\rho + O(1-\rho)] / [(1-\rho) + O\rho]$, and $h'(O) = (1-2\rho) \big/
[(1-\rho)+O\rho]^2 > 0$ for $\rho < \tfrac12$, independent of $O$.} namely
\[
\frac{P(\tilde y{=}1\mid\mathbf{x})}{P(\tilde y{=}0\mid\mathbf{x})} = h\bigl(O(\mathbf{x})\bigr),
\qquad
h(O) = \frac{\rho + O(1-\rho)}{(1-\rho) + O\rho},
\]
and $h$ is strictly increasing for every $\rho < \tfrac12$.
Training with weight ratio $w$ on the noisy labels therefore does not threshold
$O(\mathbf{x})$ at $w$, it thresholds $h(O(\mathbf{x}))$ at $w$, which is equivalent to
thresholding the \emph{true} posterior odds at
\[
\eta_{\text{true}} = h^{-1}(w) = \frac{w(1-\rho) - \rho}{(1-\rho) - w\rho}.
\]
Whatever $w$ is chosen, the classifier ends up targeting $\eta_{\text{true}}$, not $w$,
and $\eta_{\text{true}}$ depends on $\rho$, which is generally unknown. Choosing $w$ no
longer means choosing $\eta$; it means choosing $\eta$ up to an unknown,
noise-dependent distortion.
\textsc{Roll}'s objective, and AUC-optimizing objectives more generally, sidestep
this mechanism entirely: they never form a weight ratio or compare a probability to a
fixed $\eta$. They target a metric directly from whichever labels they are given.
This relies specifically on the noise being class-conditional, $\rho$ a function of $y$
alone, independent of $\mathbf{x}$, as assumed throughout \Cref{sec:noisy-labels} and in
this thesis's noise experiments. The argument above only shows that this particular
distortion has this closed form under that assumption.
This does not show that likelihood-ratio ranking actually fails
under instance-dependent noise, only that the analysis given here does not, by itself,
extend to that setting.
content/appendixs/appendix_roll_code.tex
+111 -59
View File
@@ -1,7 +1,8 @@
This appendix contains the full PyTorch implementation of the ROLL loss
functions used in the experiments. The code is reproduced verbatim from
\texttt{src/roll.py} in the accompanying implementation repository, with
unrelated losses (baselines, Beta-ROLL) omitted.
functions used in the experiments. The code is reproduced from
\texttt{src/roll.py} in the accompanying implementation repository,
reformatted for print width and with unrelated losses (baselines,
Beta-ROLL) omitted; the logic is unchanged.
\section{Gaussian ROLL}
@@ -17,8 +18,9 @@ import numpy as np
# -- helpers --
def _torch_mean_std(yh):
# MLE variance (n denominator), not torch.std's Bessel-corrected default.
yhmean = torch.mean(yh)
yhstd = torch.std(yh)
yhstd = torch.std(yh, correction=0)
return yhmean, yhstd
def _torch_normal_fit(yh):
@@ -123,6 +125,20 @@ class KernelizedROLLoss(Function):
x = torch.abs(x)
return (v * torch.exp(-v * x)) / ((1 + torch.exp(-v * x)) ** 2)
@staticmethod
def _log_sigma_prime(v, x):
"""log sigma'(x; v) = -v|x| - 2*softplus(-v|x|) [stable log-derivative].
Used wherever sigma' only ever feeds a normalised ratio (softmax):
computing sigma*(1-sigma) directly saturates to exactly 0 once
v|x| exceeds ~17 in fp32 (sigma itself rounds to exactly 1 there),
while the true derivative stays representable to ~88. This form
never materialises the raw derivative, so it has no such failure
point.
"""
vx = v * torch.abs(x)
return -vx - 2 * torch.nn.functional.softplus(-vx)
# -- KDE CDF and ICDF --
@staticmethod
@@ -131,69 +147,100 @@ class KernelizedROLLoss(Function):
return torch.mean(KernelizedROLLoss._sigma(v, tau - scores))
@staticmethod
def _icdf(v, scores, alpha, tol=1e-4, max_iters=60):
"""Find tau s.t. F_hat(tau; scores) = alpha via bisection.
def _icdf(v0, scores_neg, alpha, v1, scores_pos,
loss_tol=1e-4, max_iters=60):
"""Find tau s.t. F_hat_0(tau; scores_neg) = alpha via bisection.
Bracket: [min(scores) - 10/v, max(scores) + 10/v] always
satisfies F_hat(lo) < alpha < F_hat(hi). Converges in <=16
steps for normalised scores; 60 is a generous cap.
Bracket: lo/hi = min/max(scores_neg) + h0*log(alpha/(1-alpha)),
h0 = 1/v0. Gives exact containment F_hat_0(lo) < alpha <
F_hat_0(hi) for every alpha in (0,1), not just an asymptotic
range like a fixed +-10/v0 margin.
Stopping criterion certifies the LOSS, not just tau: F_hat_1 is
itself a CDF (monotonic), and the bracket invariant keeps the
true root tau* in [lo, hi] throughout, so F_hat_1(tau*) lies in
[F_hat_1(lo), F_hat_1(hi)] regardless of where tau* falls in the
bracket. Stop once that interval is narrower than loss_tol.
"""
if v > 10000.:
if v0 > 10000.:
logging.warning(
f'Bandwidth v={v:.1f} exceeds maximum 10000, capping')
v = 10000.
f'Bandwidth v0={v0:.1f} exceeds maximum 10000, capping')
v0 = 10000.
margin = 10.0 / v
lo = scores.min() - margin
hi = scores.max() + margin
h0 = 1.0 / v0
offset = (h0 * torch.log(alpha / (1 - alpha))
if isinstance(alpha, torch.Tensor)
else h0 * np.log(alpha / (1 - alpha)))
lo = scores_neg.min() + offset
hi = scores_neg.max() + offset
for _ in range(max_iters):
mid = (lo + hi) / 2
F_mid = KernelizedROLLoss._cdf(v, mid, scores)
if torch.abs(F_mid - alpha) < tol:
break
if F_mid < alpha:
mid = (lo + hi) / 2
F0_mid = KernelizedROLLoss._cdf(v0, mid, scores_neg)
if F0_mid < alpha:
lo = mid
else:
hi = mid
return mid
F1_lo = KernelizedROLLoss._cdf(v1, lo, scores_pos)
F1_hi = KernelizedROLLoss._cdf(v1, hi, scores_pos)
if F1_hi - F1_lo < loss_tol:
break
return (lo + hi) / 2
@staticmethod
def _icdf_batch(v, scores, alphas, tol=1e-4, max_iters=60):
"""Vectorized bisection: find tau_k s.t. F_hat(tau_k) = alpha_k.
def _icdf_batch(v0, scores_neg, alphas, v1, scores_pos,
loss_tol=1e-4, max_iters=60):
"""Vectorized bisection: find tau_k s.t. F_hat_0(tau_k) = alpha_k
for all K alphas at once.
Each iteration evaluates F_hat at all K midpoints via a single
(K, n) broadcast, replacing K sequential _icdf calls.
Each bracket uses the exact per-alpha offset
h0*log(alpha_k/(1-alpha_k)) (same containment guarantee as
_icdf) rather than a shared fixed margin. Stopping criterion is
the same certified-loss bound as _icdf, evaluated for all K
brackets at once.
"""
if float(v) > 10000.:
if float(v0) > 10000.:
logging.warning(
f'Bandwidth v={float(v):.1f} exceeds maximum 10000, capping')
v = (v.clamp(max=10000.)
if isinstance(v, torch.Tensor) else 10000.)
f'Bandwidth v0={float(v0):.1f} exceeds maximum 10000, capping')
v0 = (v0.clamp(max=10000.)
if isinstance(v0, torch.Tensor) else 10000.)
K = len(alphas)
K = len(alphas)
alphas_t = torch.tensor(alphas,
dtype=scores.dtype, device=scores.device)
margin = 10.0 / float(v)
lo = torch.full((K,), float(scores.min()) - margin,
device=scores.device)
hi = torch.full((K,), float(scores.max()) + margin,
device=scores.device)
mid = (lo + hi) / 2
dtype=scores_neg.dtype, device=scores_neg.device)
h0 = 1.0 / float(v0)
# log(alpha/(1-alpha)) computed in float64 first: casting alpha
# to the working dtype before subtracting from 1 loses all
# precision once alpha is within ~1e-7 of 1, collapsing
# 1-alpha to exactly 0 and the offset to inf.
alphas_f64 = np.asarray(alphas, dtype=np.float64)
offsets_np = h0 * np.log(alphas_f64 / (1 - alphas_f64))
offsets = torch.tensor(offsets_np, dtype=scores_neg.dtype,
device=scores_neg.device)
lo = float(scores_neg.min()) + offsets
hi = float(scores_neg.max()) + offsets
for _ in range(max_iters):
diff = mid.unsqueeze(1) - scores.unsqueeze(0) # (K, n)
F_mid = torch.mean(
KernelizedROLLoss._sigma(v, diff), dim=1) # (K,)
if torch.all(torch.abs(F_mid - alphas_t) < tol):
break
too_low = F_mid < alphas_t
lo = torch.where(too_low, mid, lo)
hi = torch.where(too_low, hi, mid)
mid = (lo + hi) / 2
mid = (lo + hi) / 2
diff0 = mid.unsqueeze(1) - scores_neg.unsqueeze(0) # (K, n0)
F0_mid = torch.mean(
KernelizedROLLoss._sigma(v0, diff0), dim=1) # (K,)
too_low = F0_mid < alphas_t
lo = torch.where(too_low, mid, lo)
hi = torch.where(too_low, hi, mid)
return mid # (K,)
diff1_lo = lo.unsqueeze(1) - scores_pos.unsqueeze(0) # (K, n1)
diff1_hi = hi.unsqueeze(1) - scores_pos.unsqueeze(0) # (K, n1)
F1_lo = torch.mean(
KernelizedROLLoss._sigma(v1, diff1_lo), dim=1) # (K,)
F1_hi = torch.mean(
KernelizedROLLoss._sigma(v1, diff1_hi), dim=1) # (K,)
if torch.all(F1_hi - F1_lo < loss_tol):
break
return (lo + hi) / 2 # (K,)
# -- forward pass --
@@ -225,10 +272,10 @@ class KernelizedROLLoss(Function):
# threshold where the negative-class CDF equals alpha_k.
if len(alphas) == 1:
taus = KernelizedROLLoss._icdf(
v0, scores_neg, alphas[0]).unsqueeze(0)
v0, scores_neg, alphas[0], v1, scores_pos).unsqueeze(0)
else:
taus = KernelizedROLLoss._icdf_batch(
v0, scores_neg, alphas) # (K,)
v0, scores_neg, alphas, v1, scores_pos) # (K,)
# loss_k = F_1_hat(tau_k) = 1 - TPR@FPR_k
diff_pos = taus.unsqueeze(1) - scores_pos.unsqueeze(0) # (K, n1)
@@ -264,12 +311,14 @@ class KernelizedROLLoss(Function):
# dF_1_hat(tau_k)/d(tau_k) = (1/n1) sum_j sigma'(tau_k - s_j^(1))
dF1_dtau = sp_pos.mean(dim=1) # (K,)
# d(tau_k)/d(s_i^(0))
diff_neg = taus.unsqueeze(1) - scores_neg.unsqueeze(0) # (K, n0)
s_neg = KernelizedROLLoss._sigma(v0, diff_neg) # (K, n0)
kernel = s_neg * (1 - s_neg) # v0 cancels
grad_tau_neg = kernel / kernel.sum( # (K, n0)
dim=1, keepdim=True).clamp(min=1e-10)
# d(tau_k)/d(s_i^(0)), computed in log-space for numerical
# stability -- replaces sigma*(1-sigma), which saturates to
# exactly 0 in fp32 once v0*diff is large, even though the true
# derivative stays representable much further out.
diff_neg = taus.unsqueeze(1) - scores_neg.unsqueeze(0) # (K, n0)
log_kernel = KernelizedROLLoss._log_sigma_prime(
v0, diff_neg) # (K, n0)
grad_tau_neg = torch.softmax(log_kernel, dim=1) # (K, n0)
# d(sum_k loss_k)/d(s_i^(0))
grad_neg = (dF1_dtau.unsqueeze(1)
@@ -288,9 +337,9 @@ class KernelizedROLLoss(Function):
# -- public constructors --
def _kernelized_roll_fpr_internal(yh, y, fpr, gamma=1.0, normalize=True,
def _kernelized_roll_fpr_internal(yh, y, cdf_target, gamma=1.0, normalize=True,
bce_weight=0.0, bce_pos_weight=1.0):
roll = KernelizedROLLoss.apply([fpr], 1, yh, y, gamma, normalize)
roll = KernelizedROLLoss.apply([cdf_target], 1, yh, y, gamma, normalize)
if bce_weight == 0.0:
return roll
pos_weight = torch.tensor([bce_pos_weight],
@@ -302,14 +351,17 @@ def _kernelized_roll_fpr_internal(yh, y, fpr, gamma=1.0, normalize=True,
def kernelized_roll_fpr(fpr, normalize=True,
bce_weight=0.0, bce_pos_weight=1.0):
# Pass 1-fpr as CDF target: F_0_hat(tau) = 1-fpr iff P(neg > tau) = fpr
return partial(_kernelized_roll_fpr_internal, fpr=1 - fpr,
return partial(_kernelized_roll_fpr_internal, cdf_target=1 - fpr,
normalize=normalize, bce_weight=bce_weight,
bce_pos_weight=bce_pos_weight)
def _kernelized_roll_tpr_internal(yh, y, tpr, gamma=1.0,
bce_weight=0.0, bce_pos_weight=1.0):
# Reduce TPR@FPR to FPR@TPR by negating scores and swapping labels
# Reduce TPR@FPR to FPR@TPR by negating scores and swapping labels.
# `1 - yh` (not `-yh`) is used here; safe only because it differs
# from `-yh` by a constant shift, and KernelizedROLLoss is
# translation-invariant in its scores.
roll = KernelizedROLLoss.apply([tpr], 1, 1 - yh, 1 - y, gamma)
if bce_weight == 0.0:
return roll
content/appendixs/appendix_score_normalization.tex
+12 -14
View File
@@ -26,7 +26,7 @@ Substituting $\tau = \tau^A/c$ into $\hat{F}_0(\tau; \tilde{s}^{(0)}, v_0^B)$:
= v_0^A\bigl(\tau^A - s_j^{(0)}\bigr)
\]
for every $j$, so $\hat{F}_0(\tau^A/c; \tilde{s}^{(0)}, v_0^B) = \hat{F}_0(\tau^A; s^{(0)}, v_0^A) = 1-\alpha$
— i.e.\ $\tau^A/c$ solves the normalised-path equation, and by uniqueness of the
i.e.\ $\tau^A/c$ solves the normalised-path equation, and by uniqueness of the
bisection root, $\tau^B = \tau^A/c$ exactly. The same substitution gives, for every $j$,
\begin{equation}
v_k^B\bigl(\tau^B - \tilde{s}_j^{(k)}\bigr) = v_k^A\bigl(\tau^A - s_j^{(k)}\bigr),
@@ -70,7 +70,7 @@ unchanged:
In both cases the outer $1/c$ from the chain rule is exactly cancelled by the $c$
introduced into $v_k^B$, confirming \Cref{prop:normalization-invariance}. (Verified
numerically to float32 precision — $\max_i|\partial L^A/\partial s_i - \partial L^B/\partial s_i|
numerically to float32 precision: $\max_i|\partial L^A/\partial s_i - \partial L^B/\partial s_i|
\approx 5.6\times10^{-9}$ on a random batch, using Silverman's rule for $\hat{h}$.)
\section{ISJ Does Not Satisfy the Hypothesis}
@@ -99,13 +99,13 @@ $10^{6}$ & $0.353732$ & $0.217671$ \\
Consequently, for a batch where ISJ (rather than the Silverman fallback) is used,
neither the loss value nor the gradient is invariant to normalisation: a directly
measured example (random batch, $n{=}10$) gives loss $0.734$ on raw scores versus
$0.767$ on normalised scores, and a maximum per-sample gradient difference of $0.014$
— both far from the exact invariance \Cref{prop:normalization-invariance} guarantees
$0.767$ on normalised scores, and a maximum per-sample gradient difference of $0.014$,
both far from the exact invariance \Cref{prop:normalization-invariance} guarantees
for Silverman. This is the genuine numerical-stability role of normalisation for the
method as actually used: not ``preventing overflow via an unchanged, merely-rescaled
argument'' (\Cref{prop:normalization-invariance} shows that mechanism is a no-op under
exact scale-equivariance), but pinning ISJ's input to the fixed scale regime — roughly
unit variance — it behaves consistently at, rather than leaving it to whatever scale the
exact scale-equivariance), but pinning ISJ's input to the fixed scale regime (roughly
unit variance) it behaves consistently at, rather than leaving it to whatever scale the
model's raw outputs happen to have at a given point in training.
\section{The Missing Chain-Rule Term for a Differentiated $\hat\sigma_\mathcal{B}$}
@@ -130,14 +130,12 @@ $\partial c/\partial s_i := 0$ by declaration, which is exactly why the implemen
gradient (\Cref{sec:roll-numerical-stability}) has no such term.
If instead $\hat\sigma_\mathcal{B}$ is differentiated through, the term in
\eqref{eq:sn-missing-term} does not generally vanish. Unlike the bandwidth case, there
is no exact-cancellation argument available here: the symmetry that would zero it out
via Euler's identity ($\sum_j \tilde{s}_j\, q_j = 0$, from $L$ being homogeneous of
degree $0$ in $\tilde{s}$) requires the bandwidth to be differentiated through the same
rescaling too — which \Cref{sec:roll-kde-backward} deliberately does not do. Numerically
(random batch, $n{=}10$, one $y_i{=}1$ sample): the frozen-$c$ gradient for that sample
is $-0.00181$ — correctly satisfying \Cref{obs:gradient-sign-correctness} ($\leq 0$
required) — while the fully-differentiated-$c$ gradient for the same sample is
\eqref{eq:sn-missing-term} does not generally vanish. There is no gaurantee
that this term does not overwhelm the first, flipping the sign and rendering the loss
useless. Numerically (random batch, $n{=}10$, one $y_i{=}1$ sample):
the frozen-$c$ gradient for that sample
is $-0.00181$, correctly satisfying \Cref{obs:gradient-sign-correctness} ($\leq 0$
required), while the fully-differentiated-$c$ gradient for the same sample is
$+0.00020$, violating it. Freezing $c$ is therefore adopted for the same reason as
freezing $v_0,v_1$ (Appendix~\ref{appendix:bandwidth-freeze}): the alternative is not
merely more complex, it can produce gradients of the wrong sign.