It's done first draft wahoo

content/discussion_and_conclusions/discussion_and_conclusions.tex

@@ -3,7 +3,7 @@
 \label{sec:discussion}
 \addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{דיון}}
 
-\subsection{Noise Robustness of \textsc{Roll}-AOC}
+\subsection{Noise Robustness of \textsc{ROLL}-AOC}
 \label{sec:discussion:noise}
 \addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{עמידות לרעש של ROLL-AOC}}
 
@@ -23,9 +23,9 @@
 % strengthening the hypothesis that this is a property of the objective rather than an artifact
 % of a specific dataset.
 
-The data shows that \textsc{Roll}-AOC is a loss robust to label noise. In KEEL, we see that under the heaviest noise rate, \textsc{Roll}-AOC degrades least. In CIFAR-10N, \textsc{Roll}-AOC suffers the least as label noise gets worse. Across other large datasets, \textsc{Roll}-AOC shows to be either competitive or leading. The experiments used a wide variety of data, both tabular and image-based, with varying levels of noise.
+The data shows that \textsc{ROLL}-AOC is a loss robust to label noise. In KEEL, we see that under the heaviest noise rate, \textsc{ROLL}-AOC degrades least. In CIFAR-10N, \textsc{ROLL}-AOC suffers the least as label noise gets worse. Across other large datasets, \textsc{ROLL}-AOC shows to be either competitive or leading. The experiments used a wide variety of data, both tabular and image-based, with varying levels of noise.
 
-Having the ability to optimize the desired metric (AUC) directly, rather than in a round about way, allows
+Having the ability to optimize the desired metric (AUC) directly, rather than in a roundabout way, allows
 our method to overcome challenging data. When using a per-sample loss, optimizing for a mislabeled sample
 might decrease the overall metric, but would do so at the cost of the performance of the model. Using AUC
 based methods allows this to be factored in already at the point of gradient weighting, so that the importance
@@ -34,7 +34,7 @@ the fitted score distribution by $1/n$, diluting its influence smoothly across t
 than producing a concentrated gradient outlier as occurs in per-sample losses.
 
 This noise-robustness pattern is consistent across all three benchmarks used in this work: KEEL (controlled
-asymmetric label noise), CIFAR-10N (real-world human annotation noise), and ChestXray14 (structural noise
+symmetric label noise), CIFAR-10N (real-world human annotation noise), and ChestXray14 (structural noise
 from NLP-extracted labels). The same advantage appearing across tabular data, natural images, and medical
 images strengthens the case that noise robustness is a structural property of the distributional objective,
 rather than an artifact of any particular dataset or noise type.
@@ -44,12 +44,12 @@ rather than an artifact of any particular dataset or noise type.
 \addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{ROLL-AOC מול GCE ו-MAE}}
 
 \textsc{GCE} and \textsc{MAE} are used specifically for their robustness to noise, yet under 20\%
-asymmetric noise on KEEL both degrade more than \textsc{Roll}-AOC (\textsc{GCE}: $-0.056$,
-\textsc{MAE}: $-0.071$, \textsc{Roll}-AOC: $-0.033$). Whether this reflects a need for domain-specific
-re-calibration or a fundamental limitation of per-sample objectives under label noise, \textsc{Roll}-AOC
+symmetric noise on KEEL both degrade more than \textsc{ROLL}-AOC (\textsc{GCE}: $-0.056$,
+\textsc{MAE}: $-0.071$, \textsc{ROLL}-AOC: $-0.033$). Whether this reflects a need for domain-specific
+re-calibration or a fundamental limitation of per-sample objectives under label noise, \textsc{ROLL}-AOC
 overcomes these challenges without any noise-specific adjustment.
 
-\subsection{\textsc{Roll}-AOC Versus \textsc{LibAUC}: Stability Under Noise}
+\subsection{\textsc{ROLL}-AOC Versus \textsc{LibAUC}: Stability Under Noise}
 \label{sec:discussion:libauc}
 \addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{ROLL-AOC מול LibAUC: יציבות תחת רעש}}
 
@@ -66,8 +66,8 @@ overcomes these challenges without any noise-specific adjustment.
 % Frame as a practical stability advantage: in noisy or heterogeneous settings,
 % PESG-based optimization may be brittle in ways that a standard Adam + KDE loss is not.
 
-While \textsc{Roll}-AOC shows improvement over per-sample losses on noisy datasets, it also outperforms
-our LibAUC baseline, which also optimizes for AUC directly.
+While \textsc{ROLL}-AOC shows improvement over per-sample losses on noisy datasets, it also outperforms
+the LibAUC baseline, which also optimizes for AUC directly.
 
 This advantage can be explained by how closely each method's loss value tracks the actual AUC.
 LibAUC replaces the intractable pairwise indicator $\mathbf{1}[f(x^+) > f(x^-)]$ with a squared surrogate:
@@ -76,15 +76,15 @@ LibAUC replaces the intractable pairwise indicator $\mathbf{1}[f(x^+) > f(x^-)]$
 \]
 The squared surrogate and the indicator it replaces follow fundamentally different patterns: the squared
 loss is sensitive to the magnitude of score differences, not only to their sign. As a result, the aggregate
-loss value gives little information about the AUC actually achieved --- a model with large score margins
+loss value gives little information about the AUC actually achieved, a model with large score margins
 on easy pairs can have a low surrogate loss yet mediocre ranking quality on hard pairs.
 
-\textsc{Roll}-AOC does not track AUC exactly either, but its error is bounded. The loss is the numerical
+\textsc{ROLL}-AOC does not track AUC exactly either, but its error is bounded. The loss is the numerical
 integral of $1 - \operatorname{TPR}(\alpha)$ over 99 FPR grid points, directly approximating the area
 over the ROC curve ($\mathrm{AOC} = 1 - \mathrm{AUC}$). As long as the KDE bandwidth is well-estimated
 and 99 thresholds provide sufficient coverage, the loss value stays close to the true AOC.
 The practical consequence is twofold: gradient weighting is more accurately aligned with ranking quality,
-and best-epoch model selection --- which picks the epoch with the lowest loss --- selects the epoch with
+and best-epoch model selection, which picks the epoch with the lowest loss, selects the epoch with
 the highest estimated AUC rather than the lowest surrogate value.
 
 \subsection{Failure of the Single-Threshold Variant (\textsc{Roll}-TPR)}
@@ -107,7 +107,7 @@ the highest estimated AUC rather than the lowest surrogate value.
 % threshold distribution and annealing toward the target operating point — or curriculum
 % strategies that introduce the fixed threshold gradually.
 
-The \textsc{Roll} framework allows for a variant that specifically optimizes for a single FPR or TPR instead of the whole curve. This would allow models to trade off performance in certain parts of the curve for performance at a given threshold. Optimization such as this is motivated. Sometimes only a specific working point is interesting, and optimizing for it directly is desired.
+The \textsc{ROLL} framework allows for a variant that specifically optimizes for a single FPR or TPR instead of the whole curve. This would allow models to trade off performance in certain parts of the curve for performance at a given threshold. Optimization such as this is motivated. Sometimes only a specific working point is interesting, and optimizing for it directly is desired.
 
 In experimentation, this did not work as well as anticipated. While on the train set, optimization was
 achieved for a certain working point, this benefit rarely translated to the test or even the validation set.
@@ -118,7 +118,7 @@ of its score space, which exacerbated already existing overfitting issues and ma
 to random initialization. On datasets such as Wisconsin and New-Thyroid1, this manifested as extreme
 run-to-run variance ($\pm 0.441$ and $\pm 0.241$ AUC respectively under clean labels), where the
 outcome was determined more by initialization than by the optimization itself.
-This stands in direct contrast to \textsc{Roll}-AOC: as noted in \Cref{rem:aoc-coverage}, the 99
+This stands in direct contrast to \textsc{ROLL}-AOC: as noted in \Cref{rem:aoc-coverage}, the 99
 simultaneously active thresholds spread gradient coverage across the full score range, making convergence
 far more robust to initialization.
 
@@ -142,7 +142,7 @@ far more robust to initialization.
 % computation and gradient derivations carry through with per-sample bandwidths with only
 % local changes to the kernel evaluation.
 
-The \textsc{Roll} framework relies on good bandwidth estimation throughout training.
+The \textsc{ROLL} framework relies on good bandwidth estimation throughout training.
 As described in \Cref{sec:kde-bandwidth}, the Improved Sheather-Jones (ISJ) estimator is the primary
 bandwidth selection method, with Silverman's rule as a closed-form fallback.
 In practice, ISJ is sensitive to small or degenerate batches early in training: when scores are not yet
@@ -175,13 +175,13 @@ can perform poorly wherever the score distribution exhibits multiple scales of v
 
 The number of FPR thresholds evaluated per forward pass is itself a trade-off. More thresholds yield a
 finer approximation of the ROC curve and, as noted in \Cref{rem:aoc-coverage}, denser gradient coverage
-across the score space --- both of which contribute to the robustness of \textsc{Roll}-AOC. Fewer
+across the score space, both of which contribute to the robustness of \textsc{ROLL}-AOC. Fewer
 thresholds reduce training cost proportionally but produce a coarser AUC estimate and sparser gradient
 signal. Each threshold evaluation requires a KDE CDF query and a bisection inversion, each $O(n)$ in
 the batch size, making the total per-batch cost $O(kn)$ where $k$ is the number of thresholds. In the
 regime used in this work ($k = 99$) the cost is manageable, but it is a genuine overhead relative to
-per-sample losses such as BCE, focal, and GCE (all $O(n)$). Treating $k$ as a hyperparameter --- for
-example, using $k = 25$ during early training and increasing it as the model stabilises --- is a
+per-sample losses such as BCE, focal, and GCE (all $O(n)$). Treating $k$ as a hyperparameter, for
+example, using $k = 25$ during early training and increasing it as the model stabilises, is a
 straightforward mitigation that has not yet been explored.
 
 % -----------------------------------------------------------------------
@@ -209,7 +209,7 @@ straightforward mitigation that has not yet been explored.
 In this work, we present \textsc{ROLL} as a general framework for differentiable, ROC-based optimization
 via parametric score distribution fitting. As the primary instantiation, we develop the KDE-based variant
 and provide its full forward pass and gradient derivations. \textsc{ROLL}-AOC directly integrates the
-estimated ROC curve over 99 evenly-spaced FPR thresholds --- a fundamentally different approach from
+estimated ROC curve over 99 evenly-spaced FPR thresholds, a fundamentally different approach from
 \textsc{LibAUC}'s pairwise formulation, which optimizes a squared surrogate over score pairs rather than
 the curve itself.
 
@@ -222,9 +222,9 @@ design objective.
 \label{sec:conclusions:limitations}
 \addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{מגבלות}}
 
-\textsc{Roll}-AOC has several practical limitations worth noting.
+\textsc{ROLL}-AOC has several practical limitations worth noting.
 The KDE instantiation uses a single bandwidth per class, which cannot accurately represent multimodal or heavy-tailed score distributions.
-Computational cost is significant: evaluating 99 FPR thresholds per forward pass makes \textsc{Roll}-AOC considerably more expensive per training step than per-sample losses or \textsc{LibAUC}'s pairwise formulation.
+Computational cost is significant: evaluating 99 FPR thresholds per forward pass makes \textsc{ROLL}-AOC considerably more expensive per training step than per-sample losses or \textsc{LibAUC}'s pairwise formulation.
 The single-threshold variant (\textsc{Roll}-TPR) is brittle in practice; gradient locality concentrates updates near one operating point and the benefit rarely transfers to held-out data.
 Finally, the framework currently lacks formal convergence guarantees; the empirical results are encouraging but not theoretically grounded.
 
@@ -242,4 +242,4 @@ Variable-bandwidth (adaptive) KDE methods address the single-bandwidth limitatio
 Formal convergence guarantees for KDE-ROLL under standard assumptions (batch size, bandwidth decay schedule, smoothness) remain an open problem, and would also clarify how the bandwidth scheduler interacts with gradient variance across training.
 
 \paragraph{Partial and weighted AUC.}
-Restricting or reweighting the FPR grid to emphasize specific operating regions is a straightforward modification to \textsc{Roll}-AOC, and a natural fit for settings such as medical imaging where only a narrow FPR range is clinically actionable.
+Restricting or reweighting the FPR grid to emphasize specific operating regions is a straightforward modification to \textsc{ROLL}-AOC, and a natural fit for settings such as medical imaging where only a narrow FPR range is clinically actionable.