diff --git a/content/appendixs/appendix_bandwidth_freeze.tex b/content/appendixs/appendix_bandwidth_freeze.tex index de9271a..c4aa26e 100644 --- a/content/appendixs/appendix_bandwidth_freeze.tex +++ b/content/appendixs/appendix_bandwidth_freeze.tex @@ -91,7 +91,7 @@ independent of the bandwidth. Silverman's rule on $\mathcal{B}_1 = \{0,b\}$ ($|\mathcal{B}_1|=2$, $\hat\sigma_1 = b/2$) gives $h_1 = \kappa b$ with $\kappa := 1.06 \cdot 2^{-1/5}/2$, i.e.\ -$v_1 = 1/(\kappa b)$ — the bandwidth scales linearly with $b$. +$v_1 = 1/(\kappa b)$: the bandwidth scales linearly with $b$. \paragraph{True adaptive loss.} Substituting $v_1 = 1/(\kappa b)$, \[ diff --git a/content/appendixs/appendix_gauss_grad.tex b/content/appendixs/appendix_gauss_grad.tex index d078b27..73e0470 100644 --- a/content/appendixs/appendix_gauss_grad.tex +++ b/content/appendixs/appendix_gauss_grad.tex @@ -1,6 +1,11 @@ We instantiate the general ROLL framework (\Cref{sec:roll-formulation}) with Gaussian-distributed scores. Each class's scores are modelled as -$\mathcal{N}(\mu_k, \sigma_k^2)$, fitted by maximum-likelihood estimation. +$\mathcal{N}(\mu_k, \sigma_k^2)$, fitted by maximum-likelihood +estimation.\footnote{If a class's scores are literally constant, $\sigma_k=0$ and the +CDF and gradient expressions throughout this appendix are undefined (division by +$\sigma_k$ or $\sigma_k^2$). This edge case is unsupported in the current +implementation; it does not arise with real, continuously-valued model outputs, but +supporting it in general would require an explicit floor on $\sigma_k$.} \paragraph{Forward pass.} @@ -27,6 +32,31 @@ Substituting into the general ROLL objective \eqref{eq:roll-tpr-at-fpr}: \label{eq:roll-tpr-at-fpr-gaussian} \end{align} +\paragraph{Closed-form AOC.} + +Unlike the general (KDE) case, the full AOC integral admits a closed form for the +Gaussian instantiation, bypassing the 99-point quadrature of \Cref{sec:roll-aoc} +entirely. Writing $\Phi$ for the standard normal CDF +($\Phi(x) = \tfrac{1}{2}[1+\operatorname{erf}(x/\sqrt{2})]$) and $\varphi$ for its +density, substitute \eqref{eq:roll-tpr-at-fpr-gaussian} into +$\mathcal{L}_{\text{AOC}} = \int_0^1 \mathcal{L}_{\text{ROLL-TPR@FPR}}(\alpha)\,d\alpha$ +and change variables via $u = \Phi^{-1}(1-\alpha)$, so $\tau = \mu_0 + \sigma_0 u$: +\[ + \mathcal{L}_{\text{AOC}}^{\text{Gaussian}} + = \int_{-\infty}^{\infty} \Phi\!\left(\frac{\mu_0-\mu_1+\sigma_0 u}{\sigma_1}\right)\varphi(u)\,du. +\] +Applying the standard identity +$\int_{-\infty}^{\infty}\Phi(a+bu)\,\varphi(u)\,du = \Phi\!\bigl(a/\sqrt{1+b^2}\bigr)$ +with $a = (\mu_0-\mu_1)/\sigma_1$, $b = \sigma_0/\sigma_1$: +\begin{equation} + \mathcal{L}_{\text{AOC}}^{\text{Gaussian}} = \Phi\!\left(\frac{\mu_0-\mu_1}{\sqrt{\sigma_0^2+\sigma_1^2}}\right). + \label{eq:gauss-aoc-closed-form} +\end{equation} +(Verified numerically against direct quadrature of the integral.) We nonetheless use +the same 99-point average as every other instantiation in this thesis, for +implementation uniformity across distribution families rather than necessity +(\Cref{sec:roll-aoc}). + \paragraph{Gradient derivation.} We derive the gradient of \eqref{eq:roll-tpr-at-fpr-gaussian} with respect to diff --git a/content/appendixs/appendix_kde_bisection.tex b/content/appendixs/appendix_kde_bisection.tex index 978a8fd..548af54 100644 --- a/content/appendixs/appendix_kde_bisection.tex +++ b/content/appendixs/appendix_kde_bisection.tex @@ -10,7 +10,8 @@ A fixed margin of $10/v_0 = 10h_0$ beyond the data range, \] only guarantees $\hat{F}_0(\mathrm{lo}) < 1-\alpha < \hat{F}_0(\mathrm{hi})$ for $1-\alpha \in (\sigma(-10;1),\,\sigma(10;1)) \approx (4.5\times10^{-5},\,1-4.5\times10^{-5})$, -since that is the range the sigmoid tails have decayed into at distance $10h_0$ — not for +since that is the range the sigmoid tails have decayed into at distance $10h_0$. This +does not hold true for every $\alpha \in (0,1)$ as a fixed margin might suggest. Instead we use an exact, target-dependent bracket. For any $p \in (0,1)$, define @@ -42,8 +43,8 @@ argument at $\max(\mathbf{s}^{(0)})$ gives $\hat{F}_0(\mathrm{hi}(p)) > p$. \end{proof} Setting $p = 1-\alpha$ gives a bracket valid for every $\alpha \in (0,1)$, unlike the -fixed-margin version above. The degenerate case $\min(\mathbf{s}^{(0)}) = \max(\mathbf{s}^{(0)})$ -— a batch with constant negative-class scores — is excluded from +fixed-margin version above. The degenerate case $\min(\mathbf{s}^{(0)}) = \max(\mathbf{s}^{(0)})$. +This represents a batch with constant negative-class scores. I is excluded from \Cref{prop:bisection-bracket} above; it also breaks the Silverman bandwidth estimate itself ($h_0 = 0$), and is addressed as part of the degenerate-batch handling still outstanding for \Cref{sec:kde-bandwidth}. @@ -68,7 +69,7 @@ certifies that the \emph{threshold} $\tau$ has been located accurately, but not of $\tau$ values can satisfy the residual bound while $\hat{F}_1(\tau)$ still varies substantially across that range. -Instead, we exploit that $\hat{F}_1$ is itself a CDF — monotonically increasing — and +Instead, we exploit that $\hat{F}_1$ is itself a CDF (monotonically increasing) and that the bracket invariant $\hat{F}_0(\mathrm{lo}) < 1-\alpha < \hat{F}_0(\mathrm{hi})$ keeps the true root $\tau^\star$ inside $[\mathrm{lo},\mathrm{hi}]$ throughout. Hence \[ diff --git a/content/appendixs/appendix_np_theory.tex b/content/appendixs/appendix_np_theory.tex new file mode 100644 index 0000000..c8e2b43 --- /dev/null +++ b/content/appendixs/appendix_np_theory.tex @@ -0,0 +1,118 @@ +This appendix gives a Neyman-Pearson-theoretic account of why order-dependent losses (objectives that depend on the ranking of scores rather than on comparing each score +to a fixed threshold) avoid a real limitation of class-weighted +per-sample losses. \Cref{sec:np-class-weighting} shows that a weighted per-sample loss + commits to a Neyman-Pearson threshold $\eta$ chosen indirectly through the class +weights. +\Cref{sec:np-noise-invariance} shows that under label noise this implicit $\eta$ is +further, unpredictably distorted. Neither limitation is specific to \textsc{Roll}: they +apply to any per-sample-weighted loss, and the way around them (targeting an operating +point or optimizing AUC directly, without ever forming a class weight or comparing to a +fixed $\eta$) applies equally to \textsc{Roll}'s TPR@FPR and ROLL-AOC objectives and to +other AUC-optimizing methods such as \textsc{LibAUC}~\cite{yuan2021auc}. + +\section{Class Weighting as an Implicit Choice of $\eta$} +\label{sec:np-class-weighting} + +Recall from \Cref{sec:neyman-pearson} that the Neyman-Pearson lemma characterizes the +optimal test at a fixed false-positive rate as thresholding the likelihood ratio +$p(\mathbf{x}\mid y{=}1)/p(\mathbf{x}\mid y{=}0)$ against some constant $\eta$: a single +threshold, fixed once for the whole test, the same for every $\mathbf{x}$. We will not +need $\eta$'s exact value below, only that it exists, that it is one constant shared by +every sample, and that a weighted per-sample loss determines it indirectly through the +class weights. + +\Cref{sec:imbalanced-tpr-fpr} noted that per-sample losses typically correct for class +imbalance by multiplying the loss on one class by a weight. This practice has a direct +reading in Neyman-Pearson terms: for a weighted per-sample loss, minimizing the pointwise +loss and thresholding the result is equivalent to thresholding the posterior +odds,\footnote{For weighted cross-entropy with weights $w_1,w_0$, minimizing +$w_1 P(y{=}1\mid\mathbf{x})(-\log q) + w_0 P(y{=}0\mid\mathbf{x})(-\log(1-q))$ over the +predicted probability $q$ gives $q^\star = w_1 P(y{=}1\mid\mathbf{x}) \big/ +\bigl[w_1 P(y{=}1\mid\mathbf{x}) + w_0 P(y{=}0\mid\mathbf{x})\bigr]$. Thresholding +$q^\star > \tfrac12$ is equivalent to $w_1 P(y{=}1\mid\mathbf{x}) > w_0 P(y{=}0\mid\mathbf{x})$, +i.e. to the threshold on $O(\mathbf{x})$ stated below.} $O(\mathbf{x}) = P(y{=}1\mid\mathbf{x})/P(y{=}0\mid\mathbf{x})$, +against a constant set by the weight ratio: +\[ + O(\mathbf{x}) > \eta, \qquad \eta = \frac{w_0}{w_1}. +\] +This exact value of $\eta$ follows from this particular (weighted cross-entropy) loss; +other weighted per-sample losses would fix it slightly differently. What matters below is +not the exact value but the qualitative fact common to all of them: $\eta$ is one constant, +the same for every sample, that moves with the chosen weight ratio. Choosing per-class +weights is therefore, knowingly or not, equivalent to choosing $\eta$ indirectly, and +without prior knowledge of the actual separability of the data. + +Concretely, for any point $\mathbf{x}$, this means the classifier ends up treating +$\mathbf{x}$ as positive or negative purely according to which side of $\eta$ its own +posterior odds $O(\mathbf{x})$ falls on: samples with $O(\mathbf{x}) > \eta$ are pushed +toward the positive class, samples with $O(\mathbf{x}) < \eta$ toward the negative class, +regardless of how separable $\mathbf{x}$ actually is from the other class. Two samples with +nearly identical posterior odds straddling $\eta$ can therefore end up on opposite sides of +the decision rule purely because of where $\eta$ happens to land, and since $\eta$ is +fixed entirely by the chosen weight ratio, it is the weight choice, not any property of the +data, that decides the fate of every such borderline sample. This is the central difference +from an order-dependent loss: a per-sample loss must commit to some $\eta$ before training +even begins, and every sample's outcome is filtered through that fixed, weight-determined +cutoff. + +Ordering-based objectives, such as our ROLL instantiations, avoid this altogether: rather +than fixing $\eta$ via class weights, they optimize for AUC or a target operating point +directly, and whatever $\eta$ this implies is discovered from the fitted score +distributions during training. No class weight, and no assumption linking class counts to +$\eta$, is needed. The same is true of any objective that optimizes AUC or a fixed +operating point directly, including \textsc{LibAUC}: none of these ever form a +class-weight ratio or compare a probability to a fixed $\eta$ in the first place. + +\section{Class Weighting Under Label Noise: An Implicit, Distorted $\eta$} +\label{sec:np-noise-invariance} + +\Cref{sec:np-class-weighting} showed that a weighted per-sample loss trained on clean +labels implicitly targets a threshold $\eta$ on the true posterior odds +$O(\mathbf{x}) = P(y{=}1\mid\mathbf{x})/P(y{=}0\mid\mathbf{x})$, set by the weight ratio. +In practice, however, training data is rarely clean. This section shows that under label +noise, the threshold actually reached is not the one chosen, regardless of how $\eta$ +itself is defined. + +Suppose labels are corrupted by class-conditional noise, the same model used in +\Cref{sec:noisy-labels} and in this thesis's noise experiments: each label is flipped with +probability $\rho \in [0,\tfrac12)$ depending only on its true class, independent of +$\mathbf{x}$. A model trained on the resulting noisy labels $\tilde y$ estimates the noisy +posterior $P(\tilde y{=}1\mid\mathbf{x})$, not the true one, so a weight ratio $w = w_0/w_1$ +now thresholds $P(\tilde y{=}1\mid\mathbf{x})/P(\tilde y{=}0\mid\mathbf{x})$, not +$O(\mathbf{x})$. These two quantities are related by a fixed function of $\rho$ +alone,\footnote{Since the noise depends only on $y$, $P(\tilde y{=}1\mid\mathbf{x}) = +(1-\rho)P(y{=}1\mid\mathbf{x}) + \rho P(y{=}0\mid\mathbf{x})$, and similarly +$P(\tilde y{=}0\mid\mathbf{x}) = \rho P(y{=}1\mid\mathbf{x}) + (1-\rho)P(y{=}0\mid\mathbf{x})$. +Dividing both by $P(y{=}0\mid\mathbf{x})$ and taking the ratio gives +$P(\tilde y{=}1\mid\mathbf{x})/P(\tilde y{=}0\mid\mathbf{x}) = h(O(\mathbf{x}))$ with +$h(O) = [\rho + O(1-\rho)] / [(1-\rho) + O\rho]$, and $h'(O) = (1-2\rho) \big/ +[(1-\rho)+O\rho]^2 > 0$ for $\rho < \tfrac12$, independent of $O$.} namely +\[ + \frac{P(\tilde y{=}1\mid\mathbf{x})}{P(\tilde y{=}0\mid\mathbf{x})} = h\bigl(O(\mathbf{x})\bigr), + \qquad + h(O) = \frac{\rho + O(1-\rho)}{(1-\rho) + O\rho}, +\] +and $h$ is strictly increasing for every $\rho < \tfrac12$. + +Training with weight ratio $w$ on the noisy labels therefore does not threshold +$O(\mathbf{x})$ at $w$, it thresholds $h(O(\mathbf{x}))$ at $w$, which is equivalent to +thresholding the \emph{true} posterior odds at +\[ + \eta_{\text{true}} = h^{-1}(w) = \frac{w(1-\rho) - \rho}{(1-\rho) - w\rho}. +\] +Whatever $w$ is chosen, the classifier ends up targeting $\eta_{\text{true}}$, not $w$, +and $\eta_{\text{true}}$ depends on $\rho$, which is generally unknown. Choosing $w$ no +longer means choosing $\eta$; it means choosing $\eta$ up to an unknown, +noise-dependent distortion. + +\textsc{Roll}'s objective, and AUC-optimizing objectives more generally, sidestep +this mechanism entirely: they never form a weight ratio or compare a probability to a +fixed $\eta$. They target a metric directly from whichever labels they are given. + +This relies specifically on the noise being class-conditional, $\rho$ a function of $y$ +alone, independent of $\mathbf{x}$, as assumed throughout \Cref{sec:noisy-labels} and in +this thesis's noise experiments. The argument above only shows that this particular +distortion has this closed form under that assumption. +This does not show that likelihood-ratio ranking actually fails +under instance-dependent noise, only that the analysis given here does not, by itself, +extend to that setting. diff --git a/content/appendixs/appendix_roll_code.tex b/content/appendixs/appendix_roll_code.tex index de616cd..e2789ce 100644 --- a/content/appendixs/appendix_roll_code.tex +++ b/content/appendixs/appendix_roll_code.tex @@ -1,7 +1,8 @@ This appendix contains the full PyTorch implementation of the ROLL loss -functions used in the experiments. The code is reproduced verbatim from -\texttt{src/roll.py} in the accompanying implementation repository, with -unrelated losses (baselines, Beta-ROLL) omitted. +functions used in the experiments. The code is reproduced from +\texttt{src/roll.py} in the accompanying implementation repository, +reformatted for print width and with unrelated losses (baselines, +Beta-ROLL) omitted; the logic is unchanged. \section{Gaussian ROLL} @@ -17,8 +18,9 @@ import numpy as np # -- helpers -- def _torch_mean_std(yh): + # MLE variance (n denominator), not torch.std's Bessel-corrected default. yhmean = torch.mean(yh) - yhstd = torch.std(yh) + yhstd = torch.std(yh, correction=0) return yhmean, yhstd def _torch_normal_fit(yh): @@ -123,6 +125,20 @@ class KernelizedROLLoss(Function): x = torch.abs(x) return (v * torch.exp(-v * x)) / ((1 + torch.exp(-v * x)) ** 2) + @staticmethod + def _log_sigma_prime(v, x): + """log sigma'(x; v) = -v|x| - 2*softplus(-v|x|) [stable log-derivative]. + + Used wherever sigma' only ever feeds a normalised ratio (softmax): + computing sigma*(1-sigma) directly saturates to exactly 0 once + v|x| exceeds ~17 in fp32 (sigma itself rounds to exactly 1 there), + while the true derivative stays representable to ~88. This form + never materialises the raw derivative, so it has no such failure + point. + """ + vx = v * torch.abs(x) + return -vx - 2 * torch.nn.functional.softplus(-vx) + # -- KDE CDF and ICDF -- @staticmethod @@ -131,69 +147,100 @@ class KernelizedROLLoss(Function): return torch.mean(KernelizedROLLoss._sigma(v, tau - scores)) @staticmethod - def _icdf(v, scores, alpha, tol=1e-4, max_iters=60): - """Find tau s.t. F_hat(tau; scores) = alpha via bisection. + def _icdf(v0, scores_neg, alpha, v1, scores_pos, + loss_tol=1e-4, max_iters=60): + """Find tau s.t. F_hat_0(tau; scores_neg) = alpha via bisection. - Bracket: [min(scores) - 10/v, max(scores) + 10/v] always - satisfies F_hat(lo) < alpha < F_hat(hi). Converges in <=16 - steps for normalised scores; 60 is a generous cap. + Bracket: lo/hi = min/max(scores_neg) + h0*log(alpha/(1-alpha)), + h0 = 1/v0. Gives exact containment F_hat_0(lo) < alpha < + F_hat_0(hi) for every alpha in (0,1), not just an asymptotic + range like a fixed +-10/v0 margin. + + Stopping criterion certifies the LOSS, not just tau: F_hat_1 is + itself a CDF (monotonic), and the bracket invariant keeps the + true root tau* in [lo, hi] throughout, so F_hat_1(tau*) lies in + [F_hat_1(lo), F_hat_1(hi)] regardless of where tau* falls in the + bracket. Stop once that interval is narrower than loss_tol. """ - if v > 10000.: + if v0 > 10000.: logging.warning( - f'Bandwidth v={v:.1f} exceeds maximum 10000, capping') - v = 10000. + f'Bandwidth v0={v0:.1f} exceeds maximum 10000, capping') + v0 = 10000. - margin = 10.0 / v - lo = scores.min() - margin - hi = scores.max() + margin + h0 = 1.0 / v0 + offset = (h0 * torch.log(alpha / (1 - alpha)) + if isinstance(alpha, torch.Tensor) + else h0 * np.log(alpha / (1 - alpha))) + lo = scores_neg.min() + offset + hi = scores_neg.max() + offset for _ in range(max_iters): - mid = (lo + hi) / 2 - F_mid = KernelizedROLLoss._cdf(v, mid, scores) - if torch.abs(F_mid - alpha) < tol: - break - if F_mid < alpha: + mid = (lo + hi) / 2 + F0_mid = KernelizedROLLoss._cdf(v0, mid, scores_neg) + if F0_mid < alpha: lo = mid else: hi = mid - return mid + F1_lo = KernelizedROLLoss._cdf(v1, lo, scores_pos) + F1_hi = KernelizedROLLoss._cdf(v1, hi, scores_pos) + if F1_hi - F1_lo < loss_tol: + break + + return (lo + hi) / 2 @staticmethod - def _icdf_batch(v, scores, alphas, tol=1e-4, max_iters=60): - """Vectorized bisection: find tau_k s.t. F_hat(tau_k) = alpha_k. + def _icdf_batch(v0, scores_neg, alphas, v1, scores_pos, + loss_tol=1e-4, max_iters=60): + """Vectorized bisection: find tau_k s.t. F_hat_0(tau_k) = alpha_k + for all K alphas at once. - Each iteration evaluates F_hat at all K midpoints via a single - (K, n) broadcast, replacing K sequential _icdf calls. + Each bracket uses the exact per-alpha offset + h0*log(alpha_k/(1-alpha_k)) (same containment guarantee as + _icdf) rather than a shared fixed margin. Stopping criterion is + the same certified-loss bound as _icdf, evaluated for all K + brackets at once. """ - if float(v) > 10000.: + if float(v0) > 10000.: logging.warning( - f'Bandwidth v={float(v):.1f} exceeds maximum 10000, capping') - v = (v.clamp(max=10000.) - if isinstance(v, torch.Tensor) else 10000.) + f'Bandwidth v0={float(v0):.1f} exceeds maximum 10000, capping') + v0 = (v0.clamp(max=10000.) + if isinstance(v0, torch.Tensor) else 10000.) - K = len(alphas) + K = len(alphas) alphas_t = torch.tensor(alphas, - dtype=scores.dtype, device=scores.device) - margin = 10.0 / float(v) - lo = torch.full((K,), float(scores.min()) - margin, - device=scores.device) - hi = torch.full((K,), float(scores.max()) + margin, - device=scores.device) - mid = (lo + hi) / 2 + dtype=scores_neg.dtype, device=scores_neg.device) + h0 = 1.0 / float(v0) + # log(alpha/(1-alpha)) computed in float64 first: casting alpha + # to the working dtype before subtracting from 1 loses all + # precision once alpha is within ~1e-7 of 1, collapsing + # 1-alpha to exactly 0 and the offset to inf. + alphas_f64 = np.asarray(alphas, dtype=np.float64) + offsets_np = h0 * np.log(alphas_f64 / (1 - alphas_f64)) + offsets = torch.tensor(offsets_np, dtype=scores_neg.dtype, + device=scores_neg.device) + lo = float(scores_neg.min()) + offsets + hi = float(scores_neg.max()) + offsets for _ in range(max_iters): - diff = mid.unsqueeze(1) - scores.unsqueeze(0) # (K, n) - F_mid = torch.mean( - KernelizedROLLoss._sigma(v, diff), dim=1) # (K,) - if torch.all(torch.abs(F_mid - alphas_t) < tol): - break - too_low = F_mid < alphas_t - lo = torch.where(too_low, mid, lo) - hi = torch.where(too_low, hi, mid) - mid = (lo + hi) / 2 + mid = (lo + hi) / 2 + diff0 = mid.unsqueeze(1) - scores_neg.unsqueeze(0) # (K, n0) + F0_mid = torch.mean( + KernelizedROLLoss._sigma(v0, diff0), dim=1) # (K,) + too_low = F0_mid < alphas_t + lo = torch.where(too_low, mid, lo) + hi = torch.where(too_low, hi, mid) - return mid # (K,) + diff1_lo = lo.unsqueeze(1) - scores_pos.unsqueeze(0) # (K, n1) + diff1_hi = hi.unsqueeze(1) - scores_pos.unsqueeze(0) # (K, n1) + F1_lo = torch.mean( + KernelizedROLLoss._sigma(v1, diff1_lo), dim=1) # (K,) + F1_hi = torch.mean( + KernelizedROLLoss._sigma(v1, diff1_hi), dim=1) # (K,) + if torch.all(F1_hi - F1_lo < loss_tol): + break + + return (lo + hi) / 2 # (K,) # -- forward pass -- @@ -225,10 +272,10 @@ class KernelizedROLLoss(Function): # threshold where the negative-class CDF equals alpha_k. if len(alphas) == 1: taus = KernelizedROLLoss._icdf( - v0, scores_neg, alphas[0]).unsqueeze(0) + v0, scores_neg, alphas[0], v1, scores_pos).unsqueeze(0) else: taus = KernelizedROLLoss._icdf_batch( - v0, scores_neg, alphas) # (K,) + v0, scores_neg, alphas, v1, scores_pos) # (K,) # loss_k = F_1_hat(tau_k) = 1 - TPR@FPR_k diff_pos = taus.unsqueeze(1) - scores_pos.unsqueeze(0) # (K, n1) @@ -264,12 +311,14 @@ class KernelizedROLLoss(Function): # dF_1_hat(tau_k)/d(tau_k) = (1/n1) sum_j sigma'(tau_k - s_j^(1)) dF1_dtau = sp_pos.mean(dim=1) # (K,) - # d(tau_k)/d(s_i^(0)) - diff_neg = taus.unsqueeze(1) - scores_neg.unsqueeze(0) # (K, n0) - s_neg = KernelizedROLLoss._sigma(v0, diff_neg) # (K, n0) - kernel = s_neg * (1 - s_neg) # v0 cancels - grad_tau_neg = kernel / kernel.sum( # (K, n0) - dim=1, keepdim=True).clamp(min=1e-10) + # d(tau_k)/d(s_i^(0)), computed in log-space for numerical + # stability -- replaces sigma*(1-sigma), which saturates to + # exactly 0 in fp32 once v0*diff is large, even though the true + # derivative stays representable much further out. + diff_neg = taus.unsqueeze(1) - scores_neg.unsqueeze(0) # (K, n0) + log_kernel = KernelizedROLLoss._log_sigma_prime( + v0, diff_neg) # (K, n0) + grad_tau_neg = torch.softmax(log_kernel, dim=1) # (K, n0) # d(sum_k loss_k)/d(s_i^(0)) grad_neg = (dF1_dtau.unsqueeze(1) @@ -288,9 +337,9 @@ class KernelizedROLLoss(Function): # -- public constructors -- -def _kernelized_roll_fpr_internal(yh, y, fpr, gamma=1.0, normalize=True, +def _kernelized_roll_fpr_internal(yh, y, cdf_target, gamma=1.0, normalize=True, bce_weight=0.0, bce_pos_weight=1.0): - roll = KernelizedROLLoss.apply([fpr], 1, yh, y, gamma, normalize) + roll = KernelizedROLLoss.apply([cdf_target], 1, yh, y, gamma, normalize) if bce_weight == 0.0: return roll pos_weight = torch.tensor([bce_pos_weight], @@ -302,14 +351,17 @@ def _kernelized_roll_fpr_internal(yh, y, fpr, gamma=1.0, normalize=True, def kernelized_roll_fpr(fpr, normalize=True, bce_weight=0.0, bce_pos_weight=1.0): # Pass 1-fpr as CDF target: F_0_hat(tau) = 1-fpr iff P(neg > tau) = fpr - return partial(_kernelized_roll_fpr_internal, fpr=1 - fpr, + return partial(_kernelized_roll_fpr_internal, cdf_target=1 - fpr, normalize=normalize, bce_weight=bce_weight, bce_pos_weight=bce_pos_weight) def _kernelized_roll_tpr_internal(yh, y, tpr, gamma=1.0, bce_weight=0.0, bce_pos_weight=1.0): - # Reduce TPR@FPR to FPR@TPR by negating scores and swapping labels + # Reduce TPR@FPR to FPR@TPR by negating scores and swapping labels. + # `1 - yh` (not `-yh`) is used here; safe only because it differs + # from `-yh` by a constant shift, and KernelizedROLLoss is + # translation-invariant in its scores. roll = KernelizedROLLoss.apply([tpr], 1, 1 - yh, 1 - y, gamma) if bce_weight == 0.0: return roll diff --git a/content/appendixs/appendix_score_normalization.tex b/content/appendixs/appendix_score_normalization.tex index d05493f..06fb0a2 100644 --- a/content/appendixs/appendix_score_normalization.tex +++ b/content/appendixs/appendix_score_normalization.tex @@ -26,7 +26,7 @@ Substituting $\tau = \tau^A/c$ into $\hat{F}_0(\tau; \tilde{s}^{(0)}, v_0^B)$: = v_0^A\bigl(\tau^A - s_j^{(0)}\bigr) \] for every $j$, so $\hat{F}_0(\tau^A/c; \tilde{s}^{(0)}, v_0^B) = \hat{F}_0(\tau^A; s^{(0)}, v_0^A) = 1-\alpha$ -— i.e.\ $\tau^A/c$ solves the normalised-path equation, and by uniqueness of the +i.e.\ $\tau^A/c$ solves the normalised-path equation, and by uniqueness of the bisection root, $\tau^B = \tau^A/c$ exactly. The same substitution gives, for every $j$, \begin{equation} v_k^B\bigl(\tau^B - \tilde{s}_j^{(k)}\bigr) = v_k^A\bigl(\tau^A - s_j^{(k)}\bigr), @@ -70,7 +70,7 @@ unchanged: In both cases the outer $1/c$ from the chain rule is exactly cancelled by the $c$ introduced into $v_k^B$, confirming \Cref{prop:normalization-invariance}. (Verified -numerically to float32 precision — $\max_i|\partial L^A/\partial s_i - \partial L^B/\partial s_i| +numerically to float32 precision: $\max_i|\partial L^A/\partial s_i - \partial L^B/\partial s_i| \approx 5.6\times10^{-9}$ on a random batch, using Silverman's rule for $\hat{h}$.) \section{ISJ Does Not Satisfy the Hypothesis} @@ -99,13 +99,13 @@ $10^{6}$ & $0.353732$ & $0.217671$ \\ Consequently, for a batch where ISJ (rather than the Silverman fallback) is used, neither the loss value nor the gradient is invariant to normalisation: a directly measured example (random batch, $n{=}10$) gives loss $0.734$ on raw scores versus -$0.767$ on normalised scores, and a maximum per-sample gradient difference of $0.014$ -— both far from the exact invariance \Cref{prop:normalization-invariance} guarantees +$0.767$ on normalised scores, and a maximum per-sample gradient difference of $0.014$, +both far from the exact invariance \Cref{prop:normalization-invariance} guarantees for Silverman. This is the genuine numerical-stability role of normalisation for the method as actually used: not ``preventing overflow via an unchanged, merely-rescaled argument'' (\Cref{prop:normalization-invariance} shows that mechanism is a no-op under -exact scale-equivariance), but pinning ISJ's input to the fixed scale regime — roughly -unit variance — it behaves consistently at, rather than leaving it to whatever scale the +exact scale-equivariance), but pinning ISJ's input to the fixed scale regime (roughly +unit variance) it behaves consistently at, rather than leaving it to whatever scale the model's raw outputs happen to have at a given point in training. \section{The Missing Chain-Rule Term for a Differentiated $\hat\sigma_\mathcal{B}$} @@ -130,14 +130,12 @@ $\partial c/\partial s_i := 0$ by declaration, which is exactly why the implemen gradient (\Cref{sec:roll-numerical-stability}) has no such term. If instead $\hat\sigma_\mathcal{B}$ is differentiated through, the term in -\eqref{eq:sn-missing-term} does not generally vanish. Unlike the bandwidth case, there -is no exact-cancellation argument available here: the symmetry that would zero it out -via Euler's identity ($\sum_j \tilde{s}_j\, q_j = 0$, from $L$ being homogeneous of -degree $0$ in $\tilde{s}$) requires the bandwidth to be differentiated through the same -rescaling too — which \Cref{sec:roll-kde-backward} deliberately does not do. Numerically -(random batch, $n{=}10$, one $y_i{=}1$ sample): the frozen-$c$ gradient for that sample -is $-0.00181$ — correctly satisfying \Cref{obs:gradient-sign-correctness} ($\leq 0$ -required) — while the fully-differentiated-$c$ gradient for the same sample is +\eqref{eq:sn-missing-term} does not generally vanish. There is no gaurantee +that this term does not overwhelm the first, flipping the sign and rendering the loss +useless. Numerically (random batch, $n{=}10$, one $y_i{=}1$ sample): +the frozen-$c$ gradient for that sample +is $-0.00181$, correctly satisfying \Cref{obs:gradient-sign-correctness} ($\leq 0$ +required), while the fully-differentiated-$c$ gradient for the same sample is $+0.00020$, violating it. Freezing $c$ is therefore adopted for the same reason as freezing $v_0,v_1$ (Appendix~\ref{appendix:bandwidth-freeze}): the alternative is not merely more complex, it can produce gradients of the wrong sign. diff --git a/content/discussion_and_conclusions/discussion_and_conclusions.tex b/content/discussion_and_conclusions/discussion_and_conclusions.tex index e91fa9e..0378cf2 100644 --- a/content/discussion_and_conclusions/discussion_and_conclusions.tex +++ b/content/discussion_and_conclusions/discussion_and_conclusions.tex @@ -3,6 +3,21 @@ \label{sec:discussion} \addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{דיון}} +\subsection{Theoretical Justification for AUC-Optimizing Losses} +\label{sec:discussion:np-justification} +\addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{הצדקה תיאורטית לפונקציות הפסד ממטבות AUC}} + +A theoretical justification for AUC-optimizing losses can be based on the implicit +assumptions made by class weighting, as interpreted through the Neyman-Pearson lemma: a +weighted per-sample loss implicitly commits to a fixed decision threshold $\eta$ chosen +indirectly through the class weights, with no reference to the operating point the task +actually requires, and this implicit threshold is further, unpredictably distorted under +label noise. Objectives that target an operating point or optimize AUC directly, +\textsc{Roll}'s TPR@FPR and ROLL-AOC objectives, as well as other AUC-optimizing methods +such as \textsc{LibAUC}~\cite{yuan2021auc}, sidestep this altogether, since none of them +ever form a class weight or compare a probability to a fixed $\eta$. The full analysis is +given in \Cref{appendix:np-theory}. + \subsection{Noise Robustness of \textsc{ROLL}-AOC} \label{sec:discussion:noise} \addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{עמידות לרעש של ROLL-AOC}} @@ -29,14 +44,12 @@ Having the ability to optimize the desired metric (AUC) directly, rather than in our method to overcome challenging data. When using a per-sample loss, optimizing for a mislabeled sample might decrease the overall metric, but would do so at the cost of the performance of the model. Using AUC based methods allows this to be factored in already at the point of gradient weighting, so that the importance -of this erred sample is factored in immediately. More precisely, in KDE-ROLL a single flipped label shifts -the fitted score distribution by $1/n$, diluting its influence smoothly across the full distribution rather -than producing a concentrated gradient outlier as occurs in per-sample losses. +of this erred sample is factored in immediately. This noise-robustness pattern is consistent across all three benchmarks used in this work: KEEL (controlled symmetric label noise), CIFAR-10N (real-world human annotation noise), and ChestXray14 (structural noise from NLP-extracted labels). The same advantage appearing across tabular data, natural images, and medical -images strengthens the case that noise robustness is a structural property of the distributional objective, +images strengthens the case that noise robustness is a property of the distributional objective, rather than an artifact of any particular dataset or noise type. \subsection{\textsc{ROLL}-AOC Versus \textsc{GCE} and \textsc{MAE}} @@ -235,8 +248,16 @@ Finally, the framework currently lacks formal convergence guarantees; the empiri \paragraph{Stabilizing single-threshold optimization.} The failure of \textsc{Roll}-TPR suggests that concentrating all gradient mass on a single operating threshold is too aggressive. A softer objective would weight each threshold's contribution according to a distribution centred on the target operating point, rather than a hard indicator at one FPR value. Unlike \textsc{LibAUC}, whose pairwise formulation has no notion of per-threshold weighting, \textsc{Roll}'s grid-based structure makes this modification straightforward. +\paragraph{Kernel-appropriate bandwidth selection.} +As noted in \Cref{sec:kde-bandwidth}, the ISJ bandwidth estimator is a plug-in +heuristic here: its optimality guarantee is derived for a Gaussian kernel, while +\textsc{Roll}-KDE uses a logistic kernel throughout. It was adopted for its ready +availability in the existing implementation, not because it is proven optimal for +this kernel. Deriving or empirically validating a bandwidth selection rule matched to +the logistic kernel specifically remains a further point of research. + \paragraph{Adaptive bandwidth estimation.} -Variable-bandwidth (adaptive) KDE methods address the single-bandwidth limitation by assigning a per-sample or per-region bandwidth inversely proportional to local density~\cite{abramson1982bandwidth,terrell1992variable}. Adapting such methods to the \textsc{Roll} framework is a natural extension: the threshold computation and gradient derivations carry through with per-sample bandwidths with only local changes to the kernel evaluation. +Variable-bandwidth (adaptive) KDE methods address the single-bandwidth limitation by assigning a per-sample or per-region bandwidth inversely proportional to local density~\cite{abramson1982bandwidth,terrell1992variable}. Adapting such methods to the \textsc{Roll} framework is a natural extension: the threshold computation and gradient derivations carry through with per-sample bandwidths with only local changes to the kernel evaluation, assuming a frozen gradient for bandwidth, as currently implemented. \paragraph{Convergence theory.} Formal convergence guarantees for KDE-ROLL under standard assumptions (batch size, bandwidth decay schedule, smoothness) remain an open problem, and would also clarify how the bandwidth scheduler interacts with gradient variance across training. diff --git a/content/experiments/experiments.tex b/content/experiments/experiments.tex index 220986a..493c842 100644 --- a/content/experiments/experiments.tex +++ b/content/experiments/experiments.tex @@ -73,7 +73,7 @@ We compare \textsc{ROLL}-AOC (the primary proposed loss, optimising the area und \item \textbf{MAE}: mean absolute error (symmetric noise-robust loss). \end{itemize} -All methods use the same network architecture, optimiser (Adam for non-LibAUC methods), and early stopping. The best model per-episode is chosen from the epoch with lowest loss. Each configuration is run for $n = 5$ independent episodes on KEEL and the tabular large datasets, $n = 3$ on CIFAR-10N and Food-101N. Reported AUC is evaluated on the test split, averaged between episodes. +All methods use the same network architecture, optimiser (Adam for non-LibAUC methods), and early stopping. The best model per-episode is chosen from the epoch with lowest loss. Each configuration is run for $n = 5$ independent episodes across all datasets. The KEEL noise experiments (\Cref{sec:results:keel}) use $n = 10$ instead: KEEL's datasets are small enough that each episode trains quickly despite the number of datasets involved, making it practical to run more episodes there for a more reliable noise-robustness comparison. Reported AUC is evaluated on the test split, averaged between episodes. \section{Results}\label{sec:experiments:results} \addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{תוצאות}} @@ -154,7 +154,7 @@ Yeast3 & \textbf{0.879{\tiny{$\pm$0.032}}} & 0.861{\tiny{$\pm$0.070}} & 0. \begin{table}[htbp] \centering -\caption{Mean test AUC ($\pm$\,std, $n=3$ episodes) on CIFAR-10N under three label-noise levels. Best per row in \textbf{bold}.} +\caption{Mean test AUC ($\pm$\,std, $n=5$ episodes) on CIFAR-10N under three label-noise levels. Best per row in \textbf{bold}.} \label{tab:results:cifar10n} \resizebox{\textwidth}{!}{% \begin{tabular}{lccccccc} @@ -182,7 +182,7 @@ On Food-101N, \textsc{ROLL}-AOC again achieves the highest AUC ($0.867\pm0.004$) \begin{table}[htbp] \centering -\caption{Mean test AUC ($\pm$\,std) on large real-world datasets ($n=5$ episodes; Food-101N and ChestXray14: $n=3$). Best per row in \textbf{bold}. ChestXray14 uses frozen DenseNet-121 embeddings; Food-101N trains end-to-end on raw pixels. LibAUC on Food-101N uses no sigmoid on model outputs.} +\caption{Mean test AUC ($\pm$\,std, $n=5$ episodes) on large real-world datasets. Best per row in \textbf{bold}. ChestXray14 uses frozen DenseNet-121 embeddings; Food-101N trains end-to-end on raw pixels. LibAUC on Food-101N uses no sigmoid on model outputs.} \label{tab:results:large} \resizebox{\textwidth}{!}{% \begin{tabular}{lccccccc} @@ -193,8 +193,8 @@ Bank Marketing & 0.931{\tiny{$\pm$0.002}} & 0.580{\tiny{$\pm$0.040}} & 0. Credit Card Fraud & \textbf{0.968{\tiny{$\pm$0.001}}} & 0.949{\tiny{$\pm$0.015}} & 0.942{\tiny{$\pm$0.020}} & 0.940{\tiny{$\pm$0.013}} & 0.948{\tiny{$\pm$0.014}} & 0.966{\tiny{$\pm$0.006}} & 0.954{\tiny{$\pm$0.009}} \\ Home Credit & \textbf{0.712{\tiny{$\pm$0.001}}} & 0.693{\tiny{$\pm$0.002}} & 0.694{\tiny{$\pm$0.002}} & 0.688{\tiny{$\pm$0.003}} & 0.698{\tiny{$\pm$0.002}} & 0.711{\tiny{$\pm$0.001}} & 0.699{\tiny{$\pm$0.004}} \\ \midrule -ChestXray14 (Effusion) & \textbf{0.839{\tiny{$\pm$0.001}}} & 0.788{\tiny{$\pm$0.008}} & 0.798{\tiny{$\pm$0.009}} & 0.729{\tiny{$\pm$0.007}} & 0.829{\tiny{$\pm$0.004}} & 0.835{\tiny{$\pm$0.002}} & 0.810{\tiny{$\pm$0.001}} \\ -Food-101N & \textbf{0.867{\tiny{$\pm$0.004}}} & 0.848{\tiny{$\pm$0.025}} & 0.847{\tiny{$\pm$0.009}} & 0.845{\tiny{$\pm$0.032}} & 0.842{\tiny{$\pm$0.006}} & 0.857{\tiny{$\pm$0.002}} & 0.855{\tiny{$\pm$0.013}} \\ +ChestXray14 (Effusion) & \textbf{0.838{\tiny{$\pm$0.001}}} & 0.790{\tiny{$\pm$0.010}} & 0.799{\tiny{$\pm$0.011}} & 0.731{\tiny{$\pm$0.008}} & 0.828{\tiny{$\pm$0.003}} & 0.836{\tiny{$\pm$0.002}} & 0.811{\tiny{$\pm$0.002}} \\ +Food-101N & \textbf{0.867{\tiny{$\pm$0.005}}} & 0.851{\tiny{$\pm$0.026}} & 0.848{\tiny{$\pm$0.010}} & 0.842{\tiny{$\pm$0.033}} & 0.844{\tiny{$\pm$0.006}} & 0.858{\tiny{$\pm$0.003}} & 0.854{\tiny{$\pm$0.012}} \\ \bottomrule \end{tabular}% } diff --git a/content/method/method.tex b/content/method/method.tex index 491f1c6..238ff98 100644 --- a/content/method/method.tex +++ b/content/method/method.tex @@ -32,7 +32,14 @@ Conversely, we may denote the inverse operations $\operatorname{FPR}^{-1}$ and $ For $\alpha \in (0,1)$, let $\tau = \operatorname{FPR}^{-1}(\alpha)$ denote the threshold achieving -$\operatorname{FPR}(\tau) = \alpha$. +$\operatorname{FPR}(\tau) = \alpha$, assuming $\operatorname{FPR}$ is continuous and strictly +decreasing in $\tau$ so that this threshold exists and is unique.\footnote{This requires the +negative-class score distribution to have no discontinuous steps. For an empirical distribution +this doesn't need to hold, but every distribution family $\operatorname{ROLL}$ is instantiated with in +this thesis (KDE with positive bandwidth (\Cref{sec:roll-kde}), Gaussian +(\Cref{appendix:gauss-grad}), and Beta (\Cref{appendix:beta-roll})) has a continuous, +strictly increasing CDF, so the statement holds for the +fitted distributions ROLL actually optimizes over.} The curve $\alpha \mapsto \operatorname{TPR}(\operatorname{FPR^{-1}}(\alpha))$ is the ROC curve. The primary learning objective is to maximize its area: \begin{equation} @@ -66,8 +73,8 @@ no threshold selection required at inference time. ROLL also supports single-threshold optimization: selecting a target $\operatorname{FPR} = \alpha_0$ and maximizing $\operatorname{TPR}$ at exactly that point. -In Neyman-Pearson terms~\cite{tong2016neyman}, this corresponds to minimizing Type-I error -at a fixed Type-II error bound. +In Neyman-Pearson terms~\cite{tong2016neyman}, this corresponds to minimizing Type-II error +at a fixed Type-I error bound. In practice, however, making these evaluations differentiable is not straightforward. @@ -104,8 +111,9 @@ TPR or FPR calculation, given a threshold, is non-differentiable. The reason for that is that the TPR and FPR as a function of $f_\theta(\mathbf{x}_i)$ are non-continuous. If $f_\theta(\mathbf{x}_i) \neq \tau$, that is, the score is not exactly on the threshold, then changing it by some $\varepsilon$ will not affect either TPR or FPR at all, and the derivative of the function w.r.t.\ $f_\theta(\mathbf{x}_i)$ will be zero. Likewise, -if $f_\theta(\mathbf{x}_i) = \tau$, the derivative of TPR/FPR w.r.t.\ $f_\theta(\mathbf{x}_i)$ becomes infinite. This derivative cannot be used -to continuously fit our model. +if $f_\theta(\mathbf{x}_i) = \tau$, the one-sided derivatives of TPR/FPR w.r.t.\ $f_\theta(\mathbf{x}_i)$ disagree: approaching from below leaves the indicator at $0$, while approaching from above sends the difference quotient to $+\infty$. The derivative at the jump is therefore \emph{undefined}, not an ordinary derivative equal to infinity, and either way there is no usable gradient to fit our model with. + +It is important to distinguish this finite-sample indicator objective from the population rate it estimates. The TPR and FPR terms above are indicator averages over the finite sample $\mathcal{D}$: each component of the sum, $\mathbf{1}[f_\theta(\mathbf{x}_i) > \tau]$, is a step function of that sample's own score, and it is this per-sample step that is non-differentiable. The distribution the dataset $\mathcal{D}$ draws from might itself have a smooth CDF, therefore an FPR or TPR term derived from said smooth CDF might be differentiable. However, in practice, we must always train off of a finite sample $\mathcal{D}$, and so the differentiability problem remains. This necessitates a different approach, to make these calculations differentiable w.r.t.\ the scores themselves, in order to fit this type of optimization into a learning platform. @@ -153,16 +161,16 @@ and extract the corresponding model score vectors: \mathbf{s}^{(0)} = \bigl(f_\theta(\mathbf{x}_i)\bigr)_{(\mathbf{x}_i,\, y_i)\,\in\,\mathcal{B}_0} \end{equation*} -We then fit a distribution family $\mathcal{F} = \{P_\phi : \phi \in \Phi\}$ separately to each +We then fit a distribution family $\mathcal{F} = \{P_\psi : \psi \in \Psi\}$ separately to each score vector in order to obtain a differentiable model of each class's score distribution: % \begin{equation*} - \hat{\phi}_1 = \operatorname{Fit}_{\mathcal{F}}\!\left(\mathbf{s}^{(1)}\right), + \hat{\psi}_1 = \operatorname{Fit}_{\mathcal{F}}\!\left(\mathbf{s}^{(1)}\right), \qquad - \hat{\phi}_0 = \operatorname{Fit}_{\mathcal{F}}\!\left(\mathbf{s}^{(0)}\right) + \hat{\psi}_0 = \operatorname{Fit}_{\mathcal{F}}\!\left(\mathbf{s}^{(0)}\right) \end{equation*} % -Let $\hat{F}_1$ and $\hat{F}_0$ denote the CDFs of $P_{\hat{\phi}_1}$ and $P_{\hat{\phi}_0}$, +Let $\hat{F}_1$ and $\hat{F}_0$ denote the CDFs of $P_{\hat{\psi}_1}$ and $P_{\hat{\psi}_0}$, respectively. The KDE instantiation is developed in \Cref{sec:roll-instantiations}; the Gaussian and Beta instantiations, including full derivations, are in \Cref{appendix:gauss-grad,appendix:beta-roll}. @@ -173,8 +181,8 @@ Using this notation, at a threshold $\tau$: \operatorname{FPR}(\tau) &= 1 - \hat{F}_0(\tau). \label{eq:fpr-cdf} \end{align} -Let $\hat{F}_1^{-1}$ and $\hat{F}_0^{-1}$ denote the quantile functions of $P_{\hat{\phi}_1}$ and -$P_{\hat{\phi}_0}$, respectively. The threshold achieving a target TPR of $\alpha$ follows directly +Let $\hat{F}_1^{-1}$ and $\hat{F}_0^{-1}$ denote the quantile functions of $P_{\hat{\psi}_1}$ and +$P_{\hat{\psi}_0}$, respectively. The threshold achieving a target TPR of $\alpha$ follows directly from \eqref{eq:tpr-cdf}: \[ \operatorname{TPR}(\tau) = \alpha \;\Longrightarrow\; 1 - \hat{F}_1(\tau) = \alpha @@ -209,7 +217,19 @@ of the two populations, and passing target rate $1-\alpha$ to the resulting TPR@FPR objective recovers the FPR@TPR objective at rate $\alpha$. All derivations and implementation details below are therefore given for \eqref{eq:roll-tpr-at-fpr} only; \eqref{eq:roll-fpr-at-tpr} follows by the same transformation applied to the inputs, with -the target rate complemented to $1-\alpha$. +the target rate complemented to $1-\alpha$.\footnote{This negation is valid whenever the +fitted distribution family is closed under score negation, as is the case for the Gaussian +and symmetric-KDE instantiations used in this thesis. It is not valid for Beta ROLL +(\Cref{appendix:beta-roll}), whose scores are constrained to $(0,1)$: negating a score +leaves this support entirely. There, the roles of the two populations are instead swapped +by replacing $f_\theta(\mathbf{x})$ with $1-f_\theta(\mathbf{x})$, which stays within +$(0,1)$ and is already computed in the Beta forward pass as the complementary score +$\mathbf{t}^{(k)} = \mathbf{1}-\mathbf{s}^{(k)}$. This substitution is exact for the same +reason the negation is exact for reflection-equivariant families: if $X\sim\operatorname{Beta}(a,b)$ +then $1-X\sim\operatorname{Beta}(b,a)$, and the log-moment estimator \eqref{eq:beta-params} +respects this swap exactly, since replacing $\mathbf{s}^{(k)}$ with $\mathbf{t}^{(k)}$ in +\eqref{eq:beta-denom}--\eqref{eq:beta-params} leaves $D_k$ unchanged and exchanges +$\hat{a}_k \leftrightarrow \hat{b}_k$.} \Cref{fig:roll-principle} provides a geometric view of \eqref{eq:roll-tpr-at-fpr}. \begin{figure}[H] @@ -264,33 +284,58 @@ The concrete form of each partial derivative depends on the choice of distributi $\mathcal{F}$; the KDE case is derived in \Cref{sec:roll-instantiations}, and the parametric cases are in \Cref{appendix:gauss-grad,appendix:beta-roll}. +\paragraph{Regularity assumptions.} The derivation above assumes +$\mathcal{B}_0, \mathcal{B}_1 \neq \emptyset$, both classes are represented in the +batch. This is not a peculiarity of ROLL: any ROC-based objective is undefined without +both classes present, since $\operatorname{TPR}$ and $\operatorname{FPR}$ are each +conditional on a single class. We enforce this operationally, via a batch sampler that +guarantees both classes appear in every batch, rather than inside the loss itself; a +batch missing a class causes an explicit error rather than a silently degraded gradient. + \subsection{ROLL-AOC: AUC Optimization} \label{sec:roll-aoc} The primary objective of ROLL is to maximize AUC directly, using the per-threshold loss \eqref{eq:roll-tpr-at-fpr} as a building block. Recall from -\Cref{sec:problem-formulation} that AUC is the integral of $\operatorname{TPR}(\alpha)$ +\Cref{sec:problem-formulation} that AUC is the integral of $\operatorname{TPR}(\operatorname{FPR}^{-1}(\alpha))$ over all FPR values: \begin{equation} - \operatorname{AUC} = \int_{0}^{1}\operatorname{TPR}(\alpha)\,d\alpha. + \operatorname{AUC} = \int_{0}^{1}\operatorname{TPR}(\operatorname{FPR}^{-1}(\alpha))\,d\alpha. \end{equation} Maximizing AUC is equivalent to minimizing the \emph{Area Over the Curve} (AOC), -$1 - \operatorname{AUC}$. Since $1 - \operatorname{TPR}(\alpha) = +$1 - \operatorname{AUC}$. Since $1 - \operatorname{TPR}(\operatorname{FPR}^{-1}(\alpha)) = \mathcal{L}_{\text{ROLL-TPR@FPR}}(\alpha)$ by definition \eqref{eq:roll-tpr-at-fpr}: \begin{equation} \mathcal{L}_{\text{AOC}} = \int_0^1\mathcal{L}_{\text{ROLL-TPR@FPR}}(\alpha)\,d\alpha. \end{equation} -Direct integration is not tractable, but the integral is well-approximated by averaging out sampled points. Evaluating at 99 evenly-spaced FPR values $\alpha = i/100$ for $i = 1, \ldots, 99$ +Direct integration is not tractable in general. An arbitrary fitted distribution admits +no closed form, but the integral is well-approximated by averaging sampled points. (The +Gaussian instantiation is an exception: it admits the exact closed form +$\mathcal{L}_{\text{AOC}}^{\text{Gaussian}} = \Phi\!\left(\frac{\mu_0-\mu_1}{\sqrt{\sigma_0^2+\sigma_1^2}}\right)$, +\Cref{appendix:gauss-grad}. We use the same 99-point quadrature uniformly across all +instantiations in this thesis for implementation consistency, not because it is +necessary for every distribution family.) Evaluating at 99 evenly-spaced FPR values +$\alpha = i/100$ for $i = 1, \ldots, 99$ (excluding the degenerate endpoints 0 and 1), the ROLL-AOC loss is: \begin{equation} - \mathcal{L}_{\text{ROLL-AOC}} = \frac{1}{99}\sum_{i=1}^{99}\mathcal{L}_{\text{ROLL-TPR@FPR}}\!\left(\frac{i}{100}\right) + \mathcal{L}_{\text{ROLL-AOC}} = \frac{1}{100}\sum_{i=1}^{99}\mathcal{L}_{\text{ROLL-TPR@FPR}}\!\left(\frac{i}{100}\right) \label{eq:roll-aoc} \end{equation} -Since $\operatorname{TPR}(\alpha)$ is monotone in $\alpha$ along the ROC curve, -$g(\alpha) := 1-\operatorname{TPR}(\alpha)$ is monotone and bounded in $[0,1]$, so -\eqref{eq:roll-aoc} approximates $\int_0^1 g(\alpha)\,d\alpha$ to within -$1/99 \approx 0.01$ by a standard Darboux-sum argument, assuming each per-threshold -term is evaluated exactly. This bound covers only the quadrature step: it does not +Since $\operatorname{TPR}(\operatorname{FPR}^{-1}(\alpha))$ is monotone in $\alpha$ along the ROC curve, +$g(\alpha) := 1-\operatorname{TPR}(\operatorname{FPR}^{-1}(\alpha))$ is monotone and bounded in $[0,1]$. +Moreover $g(1) = 1-\operatorname{TPR}(\operatorname{FPR}^{-1}(1)) = 1-\operatorname{TPR}(-\infty) = 0$ +exactly, since thresholding at $-\infty$ classifies every sample positive. This means +\eqref{eq:roll-aoc} is not merely a 99-point average standing in for the integral. It is +the exact right-Riemann sum of $\mathcal{L}_{\text{AOC}} = \int_0^1 g(\alpha)\,d\alpha$ over +$100$ equal-width subintervals of $[0,1]$, with the $100$th term, $g(1)$, known to vanish +rather than omitted. For a monotone function, the right-Riemann-sum error over $N$ equal +subintervals is bounded by the function's total range times the step size; here that gives +\[ + \bigl|\mathcal{L}_{\text{ROLL-AOC}} - \mathcal{L}_{\text{AOC}}\bigr| + \;\le\; \bigl(g(0)-g(1)\bigr)\cdot\frac{1}{100} + \;\le\; \frac{1}{100} = 0.01, +\] +using $g(0)\le 1$ and $g(1)=0$. This bound covers only the quadrature step: it does not extend to the gradient \eqref{eq:roll-aoc-gradient}, nor does it account for the gap between the KDE-fitted rates and the raw empirical scores (\Cref{sec:roll-instantiations}) or any residual numerical error in the per-threshold bisection for $\tau_k$. @@ -298,12 +343,12 @@ or any residual numerical error in the per-threshold bisection for $\tau_k$. The gradient follows by linearity of the sum: \begin{equation} \frac{\partial \mathcal{L}_{\text{ROLL-AOC}}(f_\theta(\mathbf{X}))} - {\partial f_\theta(\mathbf{x}_i)} = \frac{1}{99}\sum_{r=1}^{99}\frac{\partial \mathcal{L}_{\text{ROLL-TPR@FPR}}\!\left(f_\theta(\mathbf{X}) ; \frac{r}{100}\right)} + {\partial f_\theta(\mathbf{x}_i)} = \frac{1}{100}\sum_{r=1}^{99}\frac{\partial \mathcal{L}_{\text{ROLL-TPR@FPR}}\!\left(f_\theta(\mathbf{X}) ; \frac{r}{100}\right)} {\partial f_\theta(\mathbf{x}_i)} \label{eq:roll-aoc-gradient} \end{equation} (the summation index $r$ ranges over the $99$ thresholds; $i$, fixed throughout the sum, -indexes the sample being differentiated — distinct roles that \eqref{eq:roll-aoc} does +indexes the sample being differentiated. Distinct roles that \eqref{eq:roll-aoc} does not need to distinguish, since it has no outer per-sample index to collide with.) The concrete per-threshold gradient for the KDE instantiation is derived in \Cref{sec:roll-instantiations}; gradients for the parametric cases are in @@ -315,7 +360,7 @@ The concrete per-threshold gradient for the KDE instantiation is derived in \centering \includegraphics[width=0.62\textwidth]{content/method/figures/roll_aoc_roc.pdf} \caption{Geometric interpretation of ROLL-AOC. The ROC curve (orange) maps - each FPR value $\alpha$ to $\operatorname{TPR}(\alpha)$. + each FPR value $\alpha$ to $\operatorname{TPR}(\operatorname{FPR}^{-1}(\alpha))$. \textbf{Green}: area under the curve (AUC), the quantity we maximize. \textbf{Red}: area over the curve ($\mathrm{AOC} = 1 - \mathrm{AUC} = \mathcal{L}_{\text{ROLL-AOC}}$), @@ -596,13 +641,13 @@ For KDE-ROLL at a single operating threshold: (a) a positive-class sample's grad magnitude is the kernel PDF evaluated at its own distance to $\tau$, and vanishes as that distance grows, independent of the rest of the batch. (b) a negative-class sample's gradient magnitude is its kernel-PDF value \emph{relative to the sum over -$\mathcal{B}_0$} — samples near $\tau$ receive a larger share than those far from it +$\mathcal{B}_0$}. Samples near $\tau$ receive a larger share than those far from it \emph{within the same batch}, but this relative weighting does not by itself vanish for an individual sample merely because the whole class is far from $\tau$. \end{observation} \Cref{fig:threshold-locality} illustrates this for a simulated two-class setting in -which both classes are broadly spread around $\tau$ — the typical case, and the one in +which both classes are broadly spread around $\tau$: the typical case, and the one in which the negative class's relative weighting (\Cref{obs:gradient-locality}b) and the positive class's absolute decay (\Cref{obs:gradient-locality}a) happen to agree. The upper panel shows the score distributions and the operating threshold; the lower panel @@ -669,7 +714,7 @@ See Appendix~\ref{appendix:score-normalization}. For Silverman's rule, normalisation is therefore a pure floating-point reparametrisation: it changes neither the loss nor the gradient used for training, so it cannot be what -prevents overflow for that estimator — there is nothing left for it to change. +prevents overflow for that estimator. There is nothing left for it to change. ISJ, the primary bandwidth estimator (\Cref{sec:kde-bandwidth}), does \emph{not} satisfy the proposition's hypothesis (Appendix~\ref{appendix:score-normalization} measures its scale-dependence directly). For ISJ, both the loss value and the gradient genuinely @@ -703,23 +748,43 @@ nearest-threshold samples. We estimate the bandwidth using the Improved Sheather-Jones (ISJ) method~\cite{botev2010kde}, which minimises the asymptotic mean integrated squared error via an iterative plug-in procedure. ISJ tracks the true score distribution more -closely than closed-form rules. +closely than closed-form rules. ISJ's optimality guarantee is derived for a Gaussian +kernel, whereas the kernel used throughout this work is logistic (\Cref{sec:roll-kde}); +applying it here is a plug-in heuristic approximation, chosen for its ready +availability in the existing implementation rather than a proven-optimal fit to the +logistic kernel specifically. Kernel-appropriate bandwidth selection is left to future +work (\Cref{sec:conclusions:future}). Because ISJ is iterative, it can fail to converge on small or degenerate batches early in training. In those cases we fall back to Silverman's rule of thumb~\cite{silverman1986density}, \[ h = 1.06\,\hat{\sigma}\,n^{-1/5}, \] -which is closed-form and always produces a valid bandwidth. +which is closed-form. Silverman's rule is itself undefined at $\hat\sigma=0$. i.e.\ if a +class's scores are literally constant, which does not arise with real, continuously-valued +model outputs but is not ruled out in principle. The underlying bandwidth-selection +library substitutes a fixed fallback ($h=1$) in that case rather than propagating a +division by zero.\footnote{As a further safety net, $v=1/h$ is capped at $10{,}000$ +regardless of source. This last-resort cap technically breaks the exact +scale-equivariance assumed by \Cref{prop:normalization-invariance} once triggered, +though in practice. Especially after the numerical-stability improvements of +\Cref{sec:kde-grad-stability,appendix:bandwidth-freeze,appendix:score-normalization} +it is essentially never reached.} More generally, the ROLL derivation throughout assumes +each fitted density is nonzero at the quantile being inverted, so that the +inverse-function-theorem step of \Cref{sec:roll-kde-backward} is well-defined; the +safeguards above are exactly what keep this assumption from failing in the KDE +instantiation. The kernel parameter is then set to $v = 1/h$. \subsubsection{Bandwidth Scaling for KDE} \label{sec:kde-bandwidth-scaling} -Although we may select a valid bandwidth at each step, it tends to be -too small near the start of training, when scores are not yet -spread out and $\hat{\sigma}$ is small. This causes the KDE to undersmooth the score -distribution, producing very low gradient magnitudes and stalling convergence. +Empirically, training without a bandwidth scheduler stalls more often near the start of +training. A plausible explanation is that the classes are not yet well separated this +early: the model has not yet learned to distinguish them — so the fitted score +distributions overlap substantially relative to their spread, and the resulting +gradient carries little discriminative signal regardless of the estimated bandwidth's +absolute value. The solution was to use a bandwidth scheduler. Starting with large bandwidths resulted in gradients that spread out more evenly across all scores, and fewer non-convergence runs occurred. The bandwidth scheduler artificially scales up the bandwidth at the beginning to speed up initial training, and then tapers back to restore the intended bandwidth in order to maximise results. diff --git a/main.tex b/main.tex index 1869f05..124850c 100644 --- a/main.tex +++ b/main.tex @@ -259,6 +259,7 @@ \input{content/discussion_and_conclusions/discussion_and_conclusions} \appendix +\crefalias{chapter}{appendix} \begin{appendices} \chapter{Gaussian ROLL: Forward Pass and Gradient Derivation} @@ -316,6 +317,11 @@ \addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח יא}{\quad\qquadנרמול ציונים: שקילות, ISJ וכלל השרשרת הקפוא}}{{\beginL\thepage\endL}}} \input{content/appendixs/appendix_score_normalization} +\chapter{Theoretical Justification for AUC-Optimizing Losses via the Neyman-Pearson Lemma} +\label{appendix:np-theory} +\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח יב}{\quad\qquadהצדקה תיאורטית לפונקציות הפסד ממטבות AUC לפי למת ניימן-פירסון}}{{\beginL\thepage\endL}}} +\input{content/appendixs/appendix_np_theory} + \end{appendices} %~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%