Updated results, hebrew, abstract, intro, etc.

content/discussion_and_conclusions/discussion_and_conclusions.tex

@@ -3,7 +3,7 @@
 \label{sec:discussion}
 \addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{דיון}}
 
-\subsection{Noise Robustness of \textsc{Roll}-AoC}
+\subsection{Noise Robustness of \textsc{Roll}-AOC}
 \label{sec:discussion:noise}
 \addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{עמידות לרעש של ROLL-AOC}}
 
@@ -23,7 +23,7 @@
 % strengthening the hypothesis that this is a property of the objective rather than an artifact
 % of a specific dataset.
 
-The data shows that \textsc{Roll}-AoC is a loss robust to label noise. In KEEL, we see that under the heaviest noise rate, \textsc{Roll}-AoC degrades least. In CiFAR10n, \textsc{Roll}-AoC suffers the least as label noise gets worse. Across other large datasets, \textsc{Roll}-AoC shows to be either competitive or leading. The experiments used a wide variety of data, both tabular and image-based, with varying levels of noise.
+The data shows that \textsc{Roll}-AOC is a loss robust to label noise. In KEEL, we see that under the heaviest noise rate, \textsc{Roll}-AOC degrades least. In CIFAR-10N, \textsc{Roll}-AOC suffers the least as label noise gets worse. Across other large datasets, \textsc{Roll}-AOC shows to be either competitive or leading. The experiments used a wide variety of data, both tabular and image-based, with varying levels of noise.
 
 Having the ability to optimize the desired metric (AUC) directly, rather than in a round about way, allows
 our method to overcome challenging data. When using a per-sample loss, optimizing for a mislabeled sample
@@ -39,17 +39,17 @@ from NLP-extracted labels). The same advantage appearing across tabular data, na
 images strengthens the case that noise robustness is a structural property of the distributional objective,
 rather than an artifact of any particular dataset or noise type.
 
-\subsection{\textsc{ROLL}-AoC Versus \textsc{GCE} and \textsc{MAE}}
+\subsection{\textsc{ROLL}-AOC Versus \textsc{GCE} and \textsc{MAE}}
 \label{sec:discussion:gce-mae}
 \addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{ROLL-AOC מול GCE ו-MAE}}
 
 \textsc{GCE} and \textsc{MAE} are used specifically for their robustness to noise, yet under 20\%
-asymmetric noise on KEEL both degrade more than \textsc{Roll}-AoC (\textsc{GCE}: $-0.056$,
-\textsc{MAE}: $-0.071$, \textsc{Roll}-AoC: $-0.033$). Whether this reflects a need for domain-specific
-re-calibration or a fundamental limitation of per-sample objectives under label noise, \textsc{Roll}-AoC
+asymmetric noise on KEEL both degrade more than \textsc{Roll}-AOC (\textsc{GCE}: $-0.056$,
+\textsc{MAE}: $-0.071$, \textsc{Roll}-AOC: $-0.033$). Whether this reflects a need for domain-specific
+re-calibration or a fundamental limitation of per-sample objectives under label noise, \textsc{Roll}-AOC
 overcomes these challenges without any noise-specific adjustment.
 
-\subsection{\textsc{Roll}-AoC Versus \textsc{LibAUC}: Stability Under Noise}
+\subsection{\textsc{Roll}-AOC Versus \textsc{LibAUC}: Stability Under Noise}
 \label{sec:discussion:libauc}
 \addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{ROLL-AOC מול LibAUC: יציבות תחת רעש}}
 
@@ -66,26 +66,26 @@ overcomes these challenges without any noise-specific adjustment.
 % Frame as a practical stability advantage: in noisy or heterogeneous settings,
 % PESG-based optimization may be brittle in ways that a standard Adam + KDE loss is not.
 
-While \textsc{Roll}-AoC shows improvement over per-sample losses on noisy datasets, it also shows
-better performance than our LibAUC baseline, which also optimizes for the same metric directly.
+While \textsc{Roll}-AOC shows improvement over per-sample losses on noisy datasets, it also outperforms
+our LibAUC baseline, which also optimizes for AUC directly.
 
-This improvement could be explained by \textsc{Roll}-AoC better tracking the real AUC value.
-LibAUC's loss uses an imperfect surrogate loss. The AUC metric itself can only be acheived
-exactly when using a stepwise function instead of the pairwise surrogate loss, which is impossible to use.
-The surrogate losses used is the squared pairwise surrogate loss given:
+This advantage can be explained by how closely each method's loss value tracks the actual AUC.
+LibAUC replaces the intractable pairwise indicator $\mathbf{1}[f(x^+) > f(x^-)]$ with a squared surrogate:
+\[
+  \ell(f(x^+), f(x^-)) = \bigl(1 - f(x^+) + f(x^-)\bigr)^2
+\]
+The squared surrogate and the indicator it replaces follow fundamentally different patterns: the squared
+loss is sensitive to the magnitude of score differences, not only to their sign. As a result, the aggregate
+loss value gives little information about the AUC actually achieved --- a model with large score margins
+on easy pairs can have a low surrogate loss yet mediocre ranking quality on hard pairs.
 
-$$\ell(f(x^+), f(x^-)) = \bigl(1 - f(x^+) + f(x^-)\bigr)^2$$
-
-While this helps the math simplify in order to reduce the computation, the connection to the
-actual AUC value is lost, and so the metric isn't directly optimized for.
-
-\text{Roll}-AoC does not exactly estimate AUC either, yet the error is limited. As long as the bandwidth
-is estimated properly, and the AUC is sampled at enough rates, one can expect a value very close to the real
-AUC. These two pre-conditions are non-trivial to acheive, yet hold well enough for the experiments done as
-part of this work.
-
-This better tracking presents two benefits. Firstly, the gradients themselves are weighted in a more-exact way,
-helping our model converge towards better performance. Secondly, it presents itself in best-epoch selection. Our training process takes the model with the lowest loss. Considering our loss tracks the real AUC better (to be exact, it tracks the area over the curve which complements AUC), this allows our best model selection to be more exact.
+\textsc{Roll}-AOC does not track AUC exactly either, but its error is bounded. The loss is the numerical
+integral of $1 - \operatorname{TPR}(\alpha)$ over 99 FPR grid points, directly approximating the area
+over the ROC curve ($\mathrm{AOC} = 1 - \mathrm{AUC}$). As long as the KDE bandwidth is well-estimated
+and 99 thresholds provide sufficient coverage, the loss value stays close to the true AOC.
+The practical consequence is twofold: gradient weighting is more accurately aligned with ranking quality,
+and best-epoch model selection --- which picks the epoch with the lowest loss --- selects the epoch with
+the highest estimated AUC rather than the lowest surrogate value.
 
 \subsection{Failure of the Single-Threshold Variant (\textsc{Roll}-TPR)}
 \label{sec:discussion:roll-tpr}
@@ -118,7 +118,7 @@ of its score space, which exacerbated already existing overfitting issues and ma
 to random initialization. On datasets such as Wisconsin and New-Thyroid1, this manifested as extreme
 run-to-run variance ($\pm 0.441$ and $\pm 0.241$ AUC respectively under clean labels), where the
 outcome was determined more by initialization than by the optimization itself.
-This stands in direct contrast to \textsc{Roll}-AoC: as noted in \Cref{rem:aoc-coverage}, the 99
+This stands in direct contrast to \textsc{Roll}-AOC: as noted in \Cref{rem:aoc-coverage}, the 99
 simultaneously active thresholds spread gradient coverage across the full score range, making convergence
 far more robust to initialization.
 
@@ -175,7 +175,7 @@ can perform poorly wherever the score distribution exhibits multiple scales of v
 
 The number of FPR thresholds evaluated per forward pass is itself a trade-off. More thresholds yield a
 finer approximation of the ROC curve and, as noted in \Cref{rem:aoc-coverage}, denser gradient coverage
-across the score space --- both of which contribute to the robustness of \textsc{Roll}-AoC. Fewer
+across the score space --- both of which contribute to the robustness of \textsc{Roll}-AOC. Fewer
 thresholds reduce training cost proportionally but produce a coarser AUC estimate and sparser gradient
 signal. Each threshold evaluation requires a KDE CDF query and a bisection inversion, each $O(n)$ in
 the batch size, making the total per-batch cost $O(kn)$ where $k$ is the number of thresholds. In the
@@ -208,13 +208,13 @@ straightforward mitigation that has not yet been explored.
 
 In this work, we present \textsc{ROLL} as a general framework for differentiable, ROC-based optimization
 via parametric score distribution fitting. As the primary instantiation, we develop the KDE-based variant
-and provide its full forward pass and gradient derivations. \textsc{ROLL}-AoC directly integrates the
+and provide its full forward pass and gradient derivations. \textsc{ROLL}-AOC directly integrates the
 estimated ROC curve over 99 evenly-spaced FPR thresholds --- a fundamentally different approach from
 \textsc{LibAUC}'s pairwise formulation, which optimizes a squared surrogate over score pairs rather than
 the curve itself.
 
-We evaluated \textsc{ROLL}-AoC against losses designed for noise robustness and class imbalance. Across
-datasets spanning tabular and image data with varying noise types and rates, we show that \textsc{ROLL}-AoC
+We evaluated \textsc{ROLL}-AOC against losses designed for noise robustness and class imbalance. Across
+datasets spanning tabular and image data with varying noise types and rates, we show that \textsc{ROLL}-AOC
 handles these challenges competitively, often leading, despite noise robustness not being an explicit
 design objective.
 
@@ -222,9 +222,9 @@ design objective.
 \label{sec:conclusions:limitations}
 \addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{מגבלות}}
 
-\textsc{Roll}-AoC has several practical limitations worth noting.
+\textsc{Roll}-AOC has several practical limitations worth noting.
 The KDE instantiation uses a single bandwidth per class, which cannot accurately represent multimodal or heavy-tailed score distributions.
-Computational cost is significant: evaluating 99 FPR thresholds per forward pass makes \textsc{Roll}-AoC considerably more expensive per training step than per-sample losses or \textsc{LibAUC}'s pairwise formulation.
+Computational cost is significant: evaluating 99 FPR thresholds per forward pass makes \textsc{Roll}-AOC considerably more expensive per training step than per-sample losses or \textsc{LibAUC}'s pairwise formulation.
 The single-threshold variant (\textsc{Roll}-TPR) is brittle in practice; gradient locality concentrates updates near one operating point and the benefit rarely transfers to held-out data.
 Finally, the framework currently lacks formal convergence guarantees; the empirical results are encouraging but not theoretically grounded.
 
@@ -242,4 +242,4 @@ Variable-bandwidth (adaptive) KDE methods address the single-bandwidth limitatio
 Formal convergence guarantees for KDE-ROLL under standard assumptions (batch size, bandwidth decay schedule, smoothness) remain an open problem, and would also clarify how the bandwidth scheduler interacts with gradient variance across training.
 
 \paragraph{Partial and weighted AUC.}
-Restricting or reweighting the FPR grid to emphasize specific operating regions is a straightforward modification to \textsc{Roll}-AoC, and a natural fit for settings such as medical imaging where only a narrow FPR range is clinically actionable.
+Restricting or reweighting the FPR grid to emphasize specific operating regions is a straightforward modification to \textsc{Roll}-AOC, and a natural fit for settings such as medical imaging where only a narrow FPR range is clinically actionable.