thesis/content/discussion_and_conclusions/discussion_and_conclusions.tex

\section{Discussion}
\label{sec:discussion}
\addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{דיון}}

\subsection{Noise Robustness of \textsc{ROLL}-AOC}
\label{sec:discussion:noise}
\addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{עמידות לרעש של ROLL-AOC}}

% Restate the headline finding: ROLL-AOC degrades least under increasing label noise across all
% three benchmarks (KEEL, CIFAR-10N, ChestXray14), despite not being designed for noise robustness.
%
% Explain the mechanism: KDE distributional fitting dilutes a mislabelled sample's gradient
% contribution by 1/n. A single flipped label shifts the fitted score distribution smoothly
% rather than producing an isolated large-gradient outlier, as occurs in cross-entropy.
%
% The GCE/MAE paradox: GCE and MAE are explicitly designed for noise robustness yet degrade
% more than ROLL-AOC at 20% noise (GCE -0.056, MAE -0.071, ROLL -0.033). Discuss what this
% reveals about ranking/distributional objectives vs. per-sample objectives under label noise.
%
% Cross-benchmark consistency: the same noise-robustness pattern appears on KEEL (controlled
% asymmetric noise), CIFAR-10N (human annotation noise), and ChestXray14 (structural NLP noise),
% strengthening the hypothesis that this is a property of the objective rather than an artifact
% of a specific dataset.

The data shows that \textsc{ROLL}-AOC is a loss robust to label noise. In KEEL, we see that under the heaviest noise rate, \textsc{ROLL}-AOC degrades least. In CIFAR-10N, \textsc{ROLL}-AOC suffers the least as label noise gets worse. Across other large datasets, \textsc{ROLL}-AOC shows to be either competitive or leading. The experiments used a wide variety of data, both tabular and image-based, with varying levels of noise.

Having the ability to optimize the desired metric (AUC) directly, rather than in a roundabout way, allows
our method to overcome challenging data. When using a per-sample loss, optimizing for a mislabeled sample
might decrease the overall metric, but would do so at the cost of the performance of the model. Using AUC
based methods allows this to be factored in already at the point of gradient weighting, so that the importance
of this erred sample is factored in immediately. More precisely, in KDE-ROLL a single flipped label shifts
the fitted score distribution by $1/n$, diluting its influence smoothly across the full distribution rather
than producing a concentrated gradient outlier as occurs in per-sample losses.

This noise-robustness pattern is consistent across all three benchmarks used in this work: KEEL (controlled
symmetric label noise), CIFAR-10N (real-world human annotation noise), and ChestXray14 (structural noise
from NLP-extracted labels). The same advantage appearing across tabular data, natural images, and medical
images strengthens the case that noise robustness is a structural property of the distributional objective,
rather than an artifact of any particular dataset or noise type.

\subsection{\textsc{ROLL}-AOC Versus \textsc{GCE} and \textsc{MAE}}
\label{sec:discussion:gce-mae}
\addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{ROLL-AOC מול GCE ו-MAE}}

\textsc{GCE} and \textsc{MAE} are used specifically for their robustness to noise, yet under 20\%
symmetric noise on KEEL both degrade more than \textsc{ROLL}-AOC (\textsc{GCE}: $-0.056$,
\textsc{MAE}: $-0.071$, \textsc{ROLL}-AOC: $-0.033$). Whether this reflects a need for domain-specific
re-calibration or a fundamental limitation of per-sample objectives under label noise, \textsc{ROLL}-AOC
overcomes these challenges without any noise-specific adjustment.

\subsection{\textsc{ROLL}-AOC Versus \textsc{LibAUC}: Stability Under Noise}
\label{sec:discussion:libauc}
\addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{ROLL-AOC מול LibAUC: יציבות תחת רעש}}

% Both methods optimize AUC; ROLL-AOC is empirically more stable.
%
% Cite the evidence: LibAUC shows worst-of-all degradation at 20% noise (-0.073), was
% excluded from ChestXray14 entirely due to optimizer instability, and exhibits the highest
% variance on Haberman (0.616 ± 0.116 under clean labels).
%
% Structural explanation: LibAUC's PESG optimizer requires careful per-parameter learning
% rate tuning and is sensitive to batch statistics. KDE-ROLL's objective does not depend on
% an auxiliary optimizer state in the same way; the KDE is re-estimated fresh each batch.
%
% Frame as a practical stability advantage: in noisy or heterogeneous settings,
% PESG-based optimization may be brittle in ways that a standard Adam + KDE loss is not.

While \textsc{ROLL}-AOC shows improvement over per-sample losses on noisy datasets, it also outperforms
the LibAUC baseline, which also optimizes for AUC directly.

This advantage can be explained by how closely each method's loss value tracks the actual AUC.
LibAUC replaces the intractable pairwise indicator $\mathbf{1}[f(x^+) > f(x^-)]$ with a squared surrogate:
\[
  \ell(f(x^+), f(x^-)) = \bigl(1 - f(x^+) + f(x^-)\bigr)^2
\]
The squared surrogate and the indicator it replaces follow fundamentally different patterns: the squared
loss is sensitive to the magnitude of score differences, not only to their sign. As a result, the aggregate
loss value gives little information about the AUC actually achieved, a model with large score margins
on easy pairs can have a low surrogate loss yet mediocre ranking quality on hard pairs.

\textsc{ROLL}-AOC does not track AUC exactly either, but its error is bounded. The loss is the numerical
integral of $1 - \operatorname{TPR}(\alpha)$ over 99 FPR grid points, directly approximating the area
over the ROC curve ($\mathrm{AOC} = 1 - \mathrm{AUC}$). As long as the KDE bandwidth is well-estimated
and 99 thresholds provide sufficient coverage, the loss value stays close to the true AOC.
The practical consequence is twofold: gradient weighting is more accurately aligned with ranking quality,
and best-epoch model selection, which picks the epoch with the lowest loss, selects the epoch with
the highest estimated AUC rather than the lowest surrogate value.

\subsection{Failure of the Single-Threshold Variant (\textsc{Roll}-TPR)}
\label{sec:discussion:roll-tpr}
\addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{כישלון הגרסה ממוקדת-סף יחיד}}

% Present the failure cases: Wisconsin (±0.441 clean), New-Thyroid1 (±0.241 clean),
% severe noise degradation across KEEL.
%
% Tie back to \Cref{obs:gradient-locality}: gradient locality means a single operating
% threshold concentrates all gradient mass in a narrow score region. If that threshold is
% poorly placed early in training, the network receives almost no signal for samples
% elsewhere in the score space, allowing the model to stagnate or collapse.
%
% Contrast with \Cref{rem:aoc-coverage}: ROLL-AOC's 99 simultaneous thresholds spread
% gradient across the full score range, providing dense coverage that is more robust to
% initialization and noise perturbations.
%
% Suggested future direction: adaptive threshold scheduling — beginning with a broad
% threshold distribution and annealing toward the target operating point — or curriculum
% strategies that introduce the fixed threshold gradually.

The \textsc{ROLL} framework allows for a variant that specifically optimizes for a single FPR or TPR instead of the whole curve. This would allow models to trade off performance in certain parts of the curve for performance at a given threshold. Optimization such as this is motivated. Sometimes only a specific working point is interesting, and optimizing for it directly is desired.

In experimentation, this did not work as well as anticipated. While on the train set, optimization was
achieved for a certain working point, this benefit rarely translated to the test or even the validation set.
As established in \Cref{obs:gradient-locality}, at any training step only the samples nearest the operating
threshold contribute meaningful gradient; samples further away in score space contribute negligibly.
When the threshold is poorly placed early in training, the model receives almost no signal for the majority
of its score space, which exacerbated already existing overfitting issues and made results highly sensitive
to random initialization. On datasets such as Wisconsin and New-Thyroid1, this manifested as extreme
run-to-run variance ($\pm 0.441$ and $\pm 0.241$ AUC respectively under clean labels), where the
outcome was determined more by initialization than by the optimization itself.
This stands in direct contrast to \textsc{ROLL}-AOC: as noted in \Cref{rem:aoc-coverage}, the 99
simultaneously active thresholds spread gradient coverage across the full score range, making convergence
far more robust to initialization.

\subsection{Bandwidth: Sensitivity and Limitations}
\label{sec:discussion:bandwidth}
\addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{רוחב פס: רגישות ומגבלות}}

% The ISJ bandwidth estimator is sensitive in practice. Bandwidth scheduling (decay over
% training epochs) was necessary to achieve stable results; the tuning of this schedule
% was a significant practical difficulty.
%
% Fundamental limitation of uniform (per-class) bandwidth: a single scalar bandwidth
% assumes each class's score distribution has one characteristic scale. This assumption
% breaks for multimodal or heavy-tailed distributions, where different regions of the
% score space have different local densities.
%
% Variable bandwidth KDE — assigning a per-sample or per-region bandwidth — is a natural
% corrective. No published literature addresses variable bandwidth estimation in the context
% of AUC optimization or differentiable ranking losses; this is an open research direction.
% ROLL's KDE formulation is particularly well-suited to benefit from it: the threshold
% computation and gradient derivations carry through with per-sample bandwidths with only
% local changes to the kernel evaluation.

The \textsc{ROLL} framework relies on good bandwidth estimation throughout training.
As described in \Cref{sec:kde-bandwidth}, the Improved Sheather-Jones (ISJ) estimator is the primary
bandwidth selection method, with Silverman's rule as a closed-form fallback.
In practice, ISJ is sensitive to small or degenerate batches early in training: when scores are not yet
spread out, the iterative procedure can fail to converge, producing bandwidths that are far too small and
causing the KDE to undersmooth. A bandwidth scheduler was therefore necessary, artificially inflating the
bandwidth at the start of training and tapering it back over time so that convergence is reliable before
the estimator is trusted fully.

Even when using good kernels, KDE cannot match an empirical distribution ideally. One severe limitation is
the use of a single bandwidth for all samples. Fixed-bandwidth estimators, as used in this work,
can perform poorly wherever the score distribution exhibits multiple scales of variation.

\subsection{Computational Cost}
\label{sec:discussion:compute}
\addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{עלות חישובית}}

% ROLL-AOC evaluates 99 distinct thresholds per forward pass, each requiring a full KDE
% CDF query and threshold bisection. This is a concrete overhead relative to single-loss
% methods (BCE, focal, GCE — O(n) per batch) and even LibAUC's pairwise formulation.
%
% Characterize the overhead: wall-clock or relative training time if available; otherwise
% describe qualitatively that the per-batch cost scales with the number of FPR grid points.
%
% This is a genuine trade-off: the empirical noise robustness gains come at a computational
% cost that may be prohibitive in low-compute or latency-sensitive settings.
%
% One unexplored mitigation: treating the number of FPR grid points as a hyperparameter
% and evaluating fewer thresholds (e.g., 10 or 25 instead of 99) — the AUC approximation
% becomes coarser but training cost drops proportionally.

The number of FPR thresholds evaluated per forward pass is itself a trade-off. More thresholds yield a
finer approximation of the ROC curve and, as noted in \Cref{rem:aoc-coverage}, denser gradient coverage
across the score space, both of which contribute to the robustness of \textsc{ROLL}-AOC. Fewer
thresholds reduce training cost proportionally but produce a coarser AUC estimate and sparser gradient
signal. Each threshold evaluation requires a KDE CDF query and a bisection inversion, each $O(n)$ in
the batch size, making the total per-batch cost $O(kn)$ where $k$ is the number of thresholds. In the
regime used in this work ($k = 99$) the cost is manageable, but it is a genuine overhead relative to
per-sample losses such as BCE, focal, and GCE (all $O(n)$). Treating $k$ as a hyperparameter, for
example, using $k = 25$ during early training and increasing it as the model stabilises, is a
straightforward mitigation that has not yet been explored.

% -----------------------------------------------------------------------

\section{Conclusions}
\label{sec:conclusions}
\addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{מסקנות}}

\subsection{Summary of Contributions}
\label{sec:conclusions:contributions}
\addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{סיכום תרומות}}

% ROLL: a general framework for differentiable ROC-based optimization via parametric score
% distribution fitting, unifying the Gaussian, Beta, and KDE instantiations under one
% theoretical derivation.
%
% ROLL-AOC: the primary variant, approximating the AUC integral as a Monte Carlo average
% over 99 evenly-spaced FPR thresholds. This is distinct from LibAUC's pairwise ranking
% formulation and from partial AUC — it is a direct numerical integration of the ROC curve.
%
% Empirical demonstration of implicit noise robustness: ROLL-AOC achieves state-of-the-art
% noise robustness across tabular (KEEL), image (CIFAR-10N), and naturally noisy (ChestXray14)
% benchmarks, despite noise robustness not being an explicit design objective.

In this work, we present \textsc{ROLL} as a general framework for differentiable, ROC-based optimization
via parametric score distribution fitting. As the primary instantiation, we develop the KDE-based variant
and provide its full forward pass and gradient derivations. \textsc{ROLL}-AOC directly integrates the
estimated ROC curve over 99 evenly-spaced FPR thresholds, a fundamentally different approach from
\textsc{LibAUC}'s pairwise formulation, which optimizes a squared surrogate over score pairs rather than
the curve itself.

We evaluated \textsc{ROLL}-AOC against losses designed for noise robustness and class imbalance. Across
datasets spanning tabular and image data with varying noise types and rates, we show that \textsc{ROLL}-AOC
handles these challenges competitively, often leading, despite noise robustness not being an explicit
design objective.

\subsection{Limitations}
\label{sec:conclusions:limitations}
\addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{מגבלות}}

\textsc{ROLL}-AOC has several practical limitations worth noting.
The KDE instantiation uses a single bandwidth per class, which cannot accurately represent multimodal or heavy-tailed score distributions.
Computational cost is significant: evaluating 99 FPR thresholds per forward pass makes \textsc{ROLL}-AOC considerably more expensive per training step than per-sample losses or \textsc{LibAUC}'s pairwise formulation.
The single-threshold variant (\textsc{Roll}-TPR) is brittle in practice; gradient locality concentrates updates near one operating point and the benefit rarely transfers to held-out data.
Finally, the framework currently lacks formal convergence guarantees; the empirical results are encouraging but not theoretically grounded.

\subsection{Future Work}
\label{sec:conclusions:future}
\addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{כיוונים עתידיים}}

\paragraph{Stabilizing single-threshold optimization.}
The failure of \textsc{Roll}-TPR suggests that concentrating all gradient mass on a single operating threshold is too aggressive. A softer objective would weight each threshold's contribution according to a distribution centred on the target operating point, rather than a hard indicator at one FPR value. Unlike \textsc{LibAUC}, whose pairwise formulation has no notion of per-threshold weighting, \textsc{Roll}'s grid-based structure makes this modification straightforward.

\paragraph{Adaptive bandwidth estimation.}
Variable-bandwidth (adaptive) KDE methods address the single-bandwidth limitation by assigning a per-sample or per-region bandwidth inversely proportional to local density~\cite{abramson1982bandwidth,terrell1992variable}. Adapting such methods to the \textsc{Roll} framework is a natural extension: the threshold computation and gradient derivations carry through with per-sample bandwidths with only local changes to the kernel evaluation.

\paragraph{Convergence theory.}
Formal convergence guarantees for KDE-ROLL under standard assumptions (batch size, bandwidth decay schedule, smoothness) remain an open problem, and would also clarify how the bandwidth scheduler interacts with gradient variance across training.

\paragraph{Partial and weighted AUC.}
Restricting or reweighting the FPR grid to emphasize specific operating regions is a straightforward modification to \textsc{ROLL}-AOC, and a natural fit for settings such as medical imaging where only a narrow FPR range is clinically actionable.