% strengthening the hypothesis that this is a property of the objective rather than an artifact
% of a specific dataset.
The data shows that \textsc{Roll}-AoC is a loss robust to label noise. In KEEL, we see that under the heaviest noise rate, \textsc{Roll}-AoC degrades least. In CiFAR10n, \textsc{Roll}-AoC suffers the least as label noise gets worse. Across other large datasets, \textsc{Roll}-AoC shows to be either competitive or leading. The experiments used a wide variety of data, both tabular and image-based, with varying levels of noise.
Having the ability to optimize the desired metric (AUC) directly, rather than in a round about way, allows
our method to overcome challenging data. When using a per-sample loss, optimizing for a mislabeled sample
might decrease the overall metric, but would do so at the cost of the performance of the model. Using AUC
based methods allows this to be factored in already at the point of gradient weighting, so that the importance
of this erred sample is factored in immediately. More precisely, in KDE-ROLL a single flipped label shifts
the fitted score distribution by $1/n$, diluting its influence smoothly across the full distribution rather
than producing a concentrated gradient outlier as occurs in per-sample losses.
This noise-robustness pattern is consistent across all three benchmarks used in this work: KEEL (controlled
asymmetric label noise), CIFAR-10N (real-world human annotation noise), and ChestXray14 (structural noise
from NLP-extracted labels). The same advantage appearing across tabular data, natural images, and medical
images strengthens the case that noise robustness is a structural property of the distributional objective,
rather than an artifact of any particular dataset or noise type.
\subsection{\textsc{ROLL}-AoC Versus \textsc{GCE} and \textsc{MAE}}
\label{sec:discussion:gce-mae}
\addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{ROLL-AOC מול GCE ו-MAE}}
\textsc{GCE} and \textsc{MAE} are used specifically for their robustness to noise, yet under 20\%
asymmetric noise on KEEL both degrade more than \textsc{Roll}-AoC (\textsc{GCE}: $-0.056$,
\textsc{MAE}: $-0.071$, \textsc{Roll}-AoC: $-0.033$). Whether this reflects a need for domain-specific
re-calibration or a fundamental limitation of per-sample objectives under label noise, \textsc{Roll}-AoC
overcomes these challenges without any noise-specific adjustment.
\subsection{\textsc{Roll}-AoC Versus \textsc{LibAUC}: Stability Under Noise}
\label{sec:discussion:libauc}
\addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{ROLL-AOC מול LibAUC: יציבות תחת רעש}}
% Both methods optimize AUC; ROLL-AOC is empirically more stable.
%
% Cite the evidence: LibAUC shows worst-of-all degradation at 20% noise (-0.073), was
% excluded from ChestXray14 entirely due to optimizer instability, and exhibits the highest
% variance on Haberman (0.616 ± 0.116 under clean labels).
While this helps the math simplify in order to reduce the computation, the connection to the
actual AUC value is lost, and so the metric isn't directly optimized for.
\text{Roll}-AoC does not exactly estimate AUC either, yet the error is limited. As long as the bandwidth
is estimated properly, and the AUC is sampled at enough rates, one can expect a value very close to the real
AUC. These two pre-conditions are non-trivial to acheive, yet hold well enough for the experiments done as
part of this work.
This better tracking presents two benefits. Firstly, the gradients themselves are weighted in a more-exact way,
helping our model converge towards better performance. Secondly, it presents itself in best-epoch selection. Our training process takes the model with the lowest loss. Considering our loss tracks the real AUC better (to be exact, it tracks the area over the curve which complements AUC), this allows our best model selection to be more exact.
\subsection{Failure of the Single-Threshold Variant (\textsc{Roll}-TPR)}
\label{sec:discussion:roll-tpr}
\addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{כישלון הגרסה ממוקדת-סף יחיד}}
% Tie back to \Cref{obs:gradient-locality}: gradient locality means a single operating
% threshold concentrates all gradient mass in a narrow score region. If that threshold is
% poorly placed early in training, the network receives almost no signal for samples
% elsewhere in the score space, allowing the model to stagnate or collapse.
%
% Contrast with \Cref{rem:aoc-coverage}: ROLL-AOC's 99 simultaneous thresholds spread
% gradient across the full score range, providing dense coverage that is more robust to
% initialization and noise perturbations.
%
% Suggested future direction: adaptive threshold scheduling — beginning with a broad
% threshold distribution and annealing toward the target operating point — or curriculum
% strategies that introduce the fixed threshold gradually.
The \textsc{Roll} framework allows for a variant that specifically optimizes for a single FPR or TPR instead of the whole curve. This would allow models to trade off performance in certain parts of the curve for performance at a given threshold. Optimization such as this is motivated. Sometimes only a specific working point is interesting, and optimizing for it directly is desired.
In experimentation, this did not work as well as anticipated. While on the train set, optimization was
achieved for a certain working point, this benefit rarely translated to the test or even the validation set.
As established in \Cref{obs:gradient-locality}, at any training step only the samples nearest the operating
threshold contribute meaningful gradient; samples further away in score space contribute negligibly.
When the threshold is poorly placed early in training, the model receives almost no signal for the majority
of its score space, which exacerbated already existing overfitting issues and made results highly sensitive
to random initialization. On datasets such as Wisconsin and New-Thyroid1, this manifested as extreme
run-to-run variance ($\pm0.441$ and $\pm0.241$ AUC respectively under clean labels), where the
outcome was determined more by initialization than by the optimization itself.
This stands in direct contrast to \textsc{Roll}-AoC: as noted in \Cref{rem:aoc-coverage}, the 99
simultaneously active thresholds spread gradient coverage across the full score range, making convergence
far more robust to initialization.
\subsection{Bandwidth: Sensitivity and Limitations}
\textsc{Roll}-AoC has several practical limitations worth noting.
The KDE instantiation uses a single bandwidth per class, which cannot accurately represent multimodal or heavy-tailed score distributions.
Computational cost is significant: evaluating 99 FPR thresholds per forward pass makes \textsc{Roll}-AoC considerably more expensive per training step than per-sample losses or \textsc{LibAUC}'s pairwise formulation.
The single-threshold variant (\textsc{Roll}-TPR) is brittle in practice; gradient locality concentrates updates near one operating point and the benefit rarely transfers to held-out data.
Finally, the framework currently lacks formal convergence guarantees; the empirical results are encouraging but not theoretically grounded.
The failure of \textsc{Roll}-TPR suggests that concentrating all gradient mass on a single operating threshold is too aggressive. A softer objective would weight each threshold's contribution according to a distribution centred on the target operating point, rather than a hard indicator at one FPR value. Unlike \textsc{LibAUC}, whose pairwise formulation has no notion of per-threshold weighting, \textsc{Roll}'s grid-based structure makes this modification straightforward.
\paragraph{Adaptive bandwidth estimation.}
Variable-bandwidth (adaptive) KDE methods address the single-bandwidth limitation by assigning a per-sample or per-region bandwidth inversely proportional to local density~\cite{abramson1982bandwidth,terrell1992variable}. Adapting such methods to the \textsc{Roll} framework is a natural extension: the threshold computation and gradient derivations carry through with per-sample bandwidths with only local changes to the kernel evaluation.
\paragraph{Convergence theory.}
Formal convergence guarantees for KDE-ROLL under standard assumptions (batch size, bandwidth decay schedule, smoothness) remain an open problem, and would also clarify how the bandwidth scheduler interacts with gradient variance across training.
\paragraph{Partial and weighted AUC.}
Restricting or reweighting the FPR grid to emphasize specific operating regions is a straightforward modification to \textsc{Roll}-AoC, and a natural fit for settings such as medical imaging where only a narrow FPR range is clinically actionable.
@@ -74,7 +74,7 @@ The noise-robustness properties of ranking-based objectives are discussed in \Cr
While AUC as a metric for model performance has existed for a while, methods to optimize for it directly are relatively recent. At it's core, it is a non-trivial problem to solve. Existing efforts to directly optimize AUC have been based on the Wilcoxon-Mann-Whitney interpretation of AUC. However, directly calculating the afformentioned probability of a random positive score behing higher than a negative one introduces two complications. The first is that one must iterate over $n^2$ pairs to cover every posible pair. Additionally, the pairwise function one must evaluate ( $\mathbb{P}(f(x_1) > f(x_0))$ ) is a non-contiguous stepwise function, and is therefor on-differentiable.
The non-contiguous stepwise nature of the evaluated pairwise function is generally fixed by using a surrogate — either sigmoid-like or hinge functions are applied to the score difference.
The non-contiguous stepwise nature of the evaluated pairwise function is generally fixed by using a surrogate — either sigmoid-like or hinge functions are applied to the score difference~\cite{cortes2003auc}.
\citet{yuan2021auc} proposed a compositional stochastic gradient framework that reformulates AUC maximization as a min-max saddle-point problem, enabling mini-batch training without maintaining all pairwise scores in memory. Their PESG optimizer and the LibAUC library are the primary practical tools for deep AUC maximization and serve as the main AUC baseline in this thesis.
@@ -220,7 +220,7 @@ Some data, such as adult heights, follows a Gaussian-like distribution; the occu
events or car arrivals at an intersection follows a Poisson distribution. For arbitrary real-world data,
however, no such assumption can be made safely.
This motivates Kernel Density Estimation (KDE). Rather than assuming a parametric form, KDE uses the
This motivates Kernel Density Estimation (KDE)~\cite{parzen1962estimation}. Rather than assuming a parametric form, KDE uses the
empirical sample itself as the model: each observed point $x_i$ contributes a smooth bump, a
\emph{kernel}, to the estimated density. The assumption is that nearby points are plausibly likely,
so probability mass is spread locally around each sample. Summing all kernels and normalizing yields a
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
