aner/thesis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Updated results, hebrew, abstract, intro, etc.

Browse Source
This commit is contained in:
Aner Zakobar
2026-08-30 06:57:52 +03:00
parent 24d6fac79c
commit 275e29bde0
13 changed files with 353 additions and 268 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Main_pages/abstract.tex
+2 -2
View File
@@ -29,9 +29,9 @@
Real datasets used to train models are often plagued by
label noise, where a fraction of training labels are incorrect. Standard per-sample losses such as binary cross-entropy are sensitive to mislabeled examples and optimize per-sample accuracy rather than ranking quality, making them misaligned with the Area Under the ROC Curve (AUC), the standard metric for imbalanced classification. Noise-robust losses (GCE, MAE) address the sensitivity problem but remain per-sample and inherit the same metric mismatch. Existing AUC-maximizing methods improve on this by directly targeting ranking quality, but rely on pairwise surrogate losses that optimize for AUC indirectly.
In this work we introduce \textsc{ROLL} (Rate Optimized Likelihood Loss), a framework for differentiable ROC-curve optimization via output score distribution fitting. The primary instantiation, \textsc{ROLL}-AoC, approximates AUC as the mean TPR sampled across a range of FPR thresholds. This loss utilizes a Kernel Density Estimation to approximate the distribution of output scores at each epoch, and optimizes it directly via gradient descent with a custom backward pass. Although noise robustness is not an explicit design objective, \textsc{ROLL}-AoC exhibits strong implicit resistance to label noise.
In this work we introduce \textsc{ROLL} (Rate Optimized Likelihood Loss), a framework for differentiable ROC-curve optimization via output score distribution fitting. The primary instantiation, \textsc{ROLL}-AOC, approximates AUC as the mean TPR sampled across a range of FPR thresholds. This loss utilizes a Kernel Density Estimation to approximate the distribution of output scores at each epoch, and optimizes it directly via gradient descent with a custom backward pass. Although noise robustness is not an explicit design objective, \textsc{ROLL}-AOC exhibits strong implicit resistance to label noise.
We evaluate on twelve KEEL benchmark datasets under label noise, CIFAR-10N under real crowdworker annotation noise, three large-scale tabular datasets, and NIH ChestX-ray14 with organic NLP-extracted label noise. Across all benchmarks, \textsc{ROLL}-AoC is competitive under clean labels and leads under high noise rates, outperforming dedicated noise-robust losses.
We evaluate on twelve KEEL benchmark datasets under label noise, CIFAR-10N under real crowdworker annotation noise, three large-scale tabular datasets, and NIH ChestX-ray14 with organic NLP-extracted label noise. Across all benchmarks, \textsc{ROLL}-AOC is competitive under clean labels and leads under high noise rates, outperforming dedicated noise-robust losses.
Main_pages/abstract_heb.tex
+2 -2
View File
@@ -31,8 +31,8 @@
מערכי נתונים אמיתיים המשמשים לאימון מודלים סובלים לעיתים קרובות מרעש תוויות, שבו חלק מתוויות האימון שגויות. פונקציות הפסד סטנדרטיות המחושבות לכל דגימה בנפרד, כגון binary cross-entropy, רגישות לדגימות המתויגות באופן שגוי. הן נוטות לטייב את המודל כאשר כל דגימה נלקחת בנפרדת, במקום לטייב את דיגורן ככלל, מה שמנתק אותן מהמדד הרצוי, שטח מתחת לעקומת ROC‏ (AUC), המדד הסטנדרטי לסיווג. פונקציות הפסד עמידות לרעש (GCE ,MAE) מטפלות ברגישות זו, אך נותרות מחושבות לדגימה בודדת ויורשות את אותו פער במדד. שיטות קיימות לאופטימיזציה של AUC משפרות זאת על ידי כיוון לאיכות דירוג, אך מסתמכות על פונקציות הפסד-תחליף זוגיות ומטייבות את AUC באופן עקיף.
בעבודה זו אנו מציגים את \textenglish{\textsc{ROLL}} ‏(Rate Optimized Likelihood Loss), מסגרת לאופטימיזציה דיפרנציאבילית של עקומת ROC באמצעות התאמת התפלגות ציוני הפלט של המודל. המימוש העיקרי, \textenglish{\textsc{ROLL}-AoC}, מקרב את AUC כממוצע TPR הנדגם על פני טווח ערכי FPR. פונקציית הפסד זו משתמשת \textenglish{Kernel Density Estimation (KDE)} לקירוב התפלגות ציוני הפלט בכל שלב אימון, ומטייבת עבורה באופן ישיר. למרות שעמידות לרעש אינה מטרת עיצוב מפורשת, \textenglish{\textsc{ROLL}-AoC} מפגין עמידות משמעותית ומובהקת לרעש תוויות.
בעבודה זו אנו מציגים את \textenglish{\textsc{ROLL}} ‏(Rate Optimized Likelihood Loss), מסגרת לאופטימיזציה דיפרנציאבילית של עקומת ROC באמצעות התאמת התפלגות ציוני הפלט של המודל. המימוש העיקרי, \textenglish{\textsc{ROLL}-AOC}, מקרב את AUC כממוצע TPR הנדגם על פני טווח ערכי FPR. פונקציית הפסד זו משתמשת \textenglish{Kernel Density Estimation (KDE)} לקירוב התפלגות ציוני הפלט בכל שלב אימון, ומטייבת עבורה באופן ישיר. למרות שעמידות לרעש אינה מטרת עיצוב מפורשת, \textenglish{\textsc{ROLL}-AOC} מפגין עמידות משמעותית ומובהקת לרעש תוויות.
אנו מעריכים את השיטה על שנים-עשר מערכי נתונים מהאוסף KEEL תחת רעש תוויות, CIFAR-10N תחת רעש אנוטציה אנושי אמיתי, שלושה מערכי נתונים טבלאיים בקנה מידה גדול, ו-NIH ChestX-ray14 עם רעש תוויות אורגני שנוצר על ידי אלגוריתם עיבוד שפה טבעית. על פני כלל הבנצ'מרקים, \textenglish{\textsc{ROLL}-AoC} תחרותי בנתונים נקיים ומוביל בשיעורי רעש גבוהים, ועולה על פונקציות הפסד ייעודיות עמידות לרעש.
אנו מעריכים את השיטה על שנים-עשר מערכי נתונים מהאוסף KEEL תחת רעש תוויות, CIFAR-10N תחת רעש אנוטציה אנושי אמיתי, שלושה מערכי נתונים טבלאיים בקנה מידה גדול, ו-NIH ChestX-ray14 עם רעש תוויות אורגני שנוצר על ידי אלגוריתם עיבוד שפה טבעית. על פני כלל הבנצ'מרקים, \textenglish{\textsc{ROLL}-AOC} תחרותי בנתונים נקיים ומוביל בשיעורי רעש גבוהים, ועולה על פונקציות הפסד ייעודיות עמידות לרעש.
\selectlanguage{english}
content/appendixs/appendix_grad_balance.tex
+42
View File
@@ -0,0 +1,42 @@
This appendix gives the full proof of \Cref{prop:gradient-balance}.
\begin{proof}
We treat each class separately via a uniform-shift argument.
\medskip\noindent\textit{Sum over $\mathcal{B}_0$.}
From \Cref{eq:roll-gradient}, for $y_i = 0$:
\[
\frac{\partial \mathcal{L}}{\partial f_\theta(\mathbf{x}_i)}
= \frac{\partial \hat{F}_1(\tau)}{\partial \tau}
\cdot \frac{\partial \tau}{\partial f_\theta(\mathbf{x}_i)}
\]
We claim $\sum_{\mathbf{x}_i \in \mathcal{B}_0}\frac{\partial \tau}{\partial f_\theta(\mathbf{x}_i)} = 1$.
Shift all negative scores uniformly: $f_\theta(\mathbf{x}_i) \to f_\theta(\mathbf{x}_i) + \delta$ for
$\mathbf{x}_i \in \mathcal{B}_0$. By translation invariance, $\hat{F}_0$ simply translates, so
$\tau = \hat{F}_0^{-1}(1-\alpha)$ shifts by exactly $\delta$. Differentiating with respect to $\delta$ at
$\delta = 0$:
\[
1 = \frac{\mathrm{d}\tau}{\mathrm{d}\delta}\bigg|_{\delta=0}
= \sum_{\mathbf{x}_i \in \mathcal{B}_0} \frac{\partial \tau}{\partial f_\theta(\mathbf{x}_i)}
\]
Summing the $y_i = 0$ gradients therefore gives:
\begin{equation}
\sum_{\mathbf{x}_i \in \mathcal{B}_0}
\frac{\partial \mathcal{L}}{\partial f_\theta(\mathbf{x}_i)}
= \frac{\partial \hat{F}_1(\tau)}{\partial \tau} \cdot 1
= \frac{\partial \hat{F}_1(\tau)}{\partial \tau}
\label{eq:grad-balance-b0}
\end{equation}
\medskip\noindent\textit{Sum over $\mathcal{B}_1$.}
For $y_i = 1$, $\frac{\partial \mathcal{L}}{\partial f_\theta(\mathbf{x}_i)} = \frac{\partial \hat{F}_1(\tau)}{\partial f_\theta(\mathbf{x}_i)}$.
Shifting all positive scores by $\delta$ and applying translation invariance of $\hat{F}_1$:
$\hat{F}_1(\tau\,;\,\mathbf{s}+\delta\mathbf{1}) = \hat{F}_1(\tau-\delta\,;\,\mathbf{s})$.
Differentiating at $\delta = 0$:
\[
\sum_{\mathbf{x}_i \in \mathcal{B}_1}
\frac{\partial \hat{F}_1(\tau)}{\partial f_\theta(\mathbf{x}_i)}
= -\frac{\partial \hat{F}_1(\tau)}{\partial \tau}
\]
Combining with \eqref{eq:grad-balance-b0} gives \eqref{eq:grad-balance}.
\end{proof}
content/appendixs/appendix_keel_noise.tex
+54
View File
@@ -0,0 +1,54 @@
The tables below report mean test AUC at $\varepsilon = 5\%$ and $\varepsilon = 10\%$ asymmetric label noise on the KEEL benchmark.
Results at $\varepsilon = 20\%$ noise, where differentiation between methods is strongest, are discussed and tabled in the main text (\Cref{sec:results:keel}).
\begin{table}[htbp]
\centering
\caption{Mean test AUC ($\pm$\,std, $n=10$ episodes) on KEEL datasets at $\epsilon=5\%$ asymmetric label noise. Best per row in \textbf{bold}.}
\label{tab:results:keel-noise-05}
\resizebox{\textwidth}{!}{%
\begin{tabular}{lccccccc}
\toprule
Dataset & \textsc{Roll}-AOC & BCE-W & Focal & Asym. & GCE & LibAUC & MAE \\
\midrule
Glass0 & 0.797{\tiny{$\pm$0.046}} & 0.786{\tiny{$\pm$0.019}} & \textbf{0.806{\tiny{$\pm$0.030}}} & 0.792{\tiny{$\pm$0.041}} & 0.794{\tiny{$\pm$0.029}} & 0.791{\tiny{$\pm$0.056}} & 0.792{\tiny{$\pm$0.058}} \\
Glass1 & 0.728{\tiny{$\pm$0.037}} & 0.736{\tiny{$\pm$0.032}} & 0.730{\tiny{$\pm$0.074}} & 0.704{\tiny{$\pm$0.028}} & 0.717{\tiny{$\pm$0.059}} & 0.694{\tiny{$\pm$0.060}} & \textbf{0.738{\tiny{$\pm$0.047}}} \\
Glass2 & 0.677{\tiny{$\pm$0.095}} & 0.665{\tiny{$\pm$0.128}} & 0.581{\tiny{$\pm$0.091}} & 0.662{\tiny{$\pm$0.050}} & 0.651{\tiny{$\pm$0.078}} & 0.614{\tiny{$\pm$0.085}} & \textbf{0.686{\tiny{$\pm$0.091}}} \\
Glass6 & 0.972{\tiny{$\pm$0.021}} & 0.972{\tiny{$\pm$0.035}} & 0.978{\tiny{$\pm$0.016}} & \textbf{0.981{\tiny{$\pm$0.016}}} & 0.947{\tiny{$\pm$0.049}} & 0.967{\tiny{$\pm$0.036}} & 0.971{\tiny{$\pm$0.031}} \\
Haberman & 0.666{\tiny{$\pm$0.094}} & 0.627{\tiny{$\pm$0.124}} & 0.694{\tiny{$\pm$0.111}} & 0.693{\tiny{$\pm$0.065}} & 0.687{\tiny{$\pm$0.099}} & 0.651{\tiny{$\pm$0.125}} & \textbf{0.716{\tiny{$\pm$0.055}}} \\
Iris0 & 0.990{\tiny{$\pm$0.014}} & 0.998{\tiny{$\pm$0.006}} & 0.999{\tiny{$\pm$0.002}} & 0.999{\tiny{$\pm$0.002}} & \textbf{1.000{\tiny{$\pm$0.000}}} & \textbf{1.000{\tiny{$\pm$0.000}}} & 0.999{\tiny{$\pm$0.003}} \\
New-Thyroid1 & 0.984{\tiny{$\pm$0.010}} & 0.986{\tiny{$\pm$0.007}} & 0.971{\tiny{$\pm$0.046}} & 0.989{\tiny{$\pm$0.006}} & 0.987{\tiny{$\pm$0.019}} & 0.983{\tiny{$\pm$0.023}} & \textbf{0.991{\tiny{$\pm$0.012}}} \\
Pima & 0.780{\tiny{$\pm$0.021}} & 0.784{\tiny{$\pm$0.016}} & 0.780{\tiny{$\pm$0.019}} & 0.767{\tiny{$\pm$0.023}} & \textbf{0.787{\tiny{$\pm$0.011}}} & 0.756{\tiny{$\pm$0.023}} & 0.775{\tiny{$\pm$0.023}} \\
Vehicle2 & 0.975{\tiny{$\pm$0.007}} & \textbf{0.977{\tiny{$\pm$0.006}}} & 0.974{\tiny{$\pm$0.010}} & 0.969{\tiny{$\pm$0.016}} & 0.976{\tiny{$\pm$0.007}} & 0.975{\tiny{$\pm$0.008}} & 0.976{\tiny{$\pm$0.009}} \\
Vowel0 & 0.988{\tiny{$\pm$0.012}} & 0.981{\tiny{$\pm$0.019}} & 0.989{\tiny{$\pm$0.017}} & \textbf{0.991{\tiny{$\pm$0.006}}} & 0.985{\tiny{$\pm$0.014}} & 0.973{\tiny{$\pm$0.025}} & 0.978{\tiny{$\pm$0.014}} \\
Wisconsin & 0.998{\tiny{$\pm$0.002}} & 0.998{\tiny{$\pm$0.001}} & \textbf{0.999{\tiny{$\pm$0.001}}} & \textbf{0.999{\tiny{$\pm$0.001}}} & 0.998{\tiny{$\pm$0.001}} & 0.998{\tiny{$\pm$0.001}} & 0.998{\tiny{$\pm$0.001}} \\
Yeast3 & 0.900{\tiny{$\pm$0.020}} & 0.914{\tiny{$\pm$0.028}} & 0.915{\tiny{$\pm$0.034}} & \textbf{0.916{\tiny{$\pm$0.026}}} & \textbf{0.916{\tiny{$\pm$0.032}}} & 0.908{\tiny{$\pm$0.029}} & 0.909{\tiny{$\pm$0.024}} \\
\bottomrule
\end{tabular}%
}
\end{table}
\begin{table}[htbp]
\centering
\caption{Mean test AUC ($\pm$\,std, $n=10$ episodes) on KEEL datasets at $\epsilon=10\%$ asymmetric label noise. Best per row in \textbf{bold}.}
\label{tab:results:keel-noise-10}
\resizebox{\textwidth}{!}{%
\begin{tabular}{lccccccc}
\toprule
Dataset & \textsc{Roll}-AOC & BCE-W & Focal & Asym. & GCE & LibAUC & MAE \\
\midrule
Glass0 & 0.773{\tiny{$\pm$0.070}} & 0.789{\tiny{$\pm$0.037}} & 0.771{\tiny{$\pm$0.037}} & \textbf{0.821{\tiny{$\pm$0.026}}} & 0.786{\tiny{$\pm$0.057}} & 0.800{\tiny{$\pm$0.032}} & 0.807{\tiny{$\pm$0.050}} \\
Glass1 & 0.719{\tiny{$\pm$0.040}} & 0.716{\tiny{$\pm$0.079}} & 0.718{\tiny{$\pm$0.056}} & 0.710{\tiny{$\pm$0.049}} & 0.729{\tiny{$\pm$0.048}} & 0.675{\tiny{$\pm$0.070}} & \textbf{0.738{\tiny{$\pm$0.060}}} \\
Glass2 & 0.695{\tiny{$\pm$0.075}} & 0.644{\tiny{$\pm$0.072}} & 0.687{\tiny{$\pm$0.127}} & 0.655{\tiny{$\pm$0.048}} & \textbf{0.713{\tiny{$\pm$0.107}}} & 0.637{\tiny{$\pm$0.093}} & 0.673{\tiny{$\pm$0.111}} \\
Glass6 & 0.947{\tiny{$\pm$0.050}} & \textbf{0.973{\tiny{$\pm$0.026}}} & 0.950{\tiny{$\pm$0.041}} & 0.971{\tiny{$\pm$0.024}} & 0.938{\tiny{$\pm$0.047}} & 0.937{\tiny{$\pm$0.048}} & 0.935{\tiny{$\pm$0.054}} \\
Haberman & 0.638{\tiny{$\pm$0.098}} & 0.700{\tiny{$\pm$0.074}} & 0.652{\tiny{$\pm$0.106}} & 0.658{\tiny{$\pm$0.087}} & 0.636{\tiny{$\pm$0.112}} & 0.694{\tiny{$\pm$0.129}} & \textbf{0.732{\tiny{$\pm$0.022}}} \\
Iris0 & 0.991{\tiny{$\pm$0.020}} & 0.950{\tiny{$\pm$0.158}} & 0.950{\tiny{$\pm$0.158}} & \textbf{1.000{\tiny{$\pm$0.000}}} & \textbf{1.000{\tiny{$\pm$0.001}}} & \textbf{1.000{\tiny{$\pm$0.000}}} & \textbf{1.000{\tiny{$\pm$0.000}}} \\
New-Thyroid1 & 0.986{\tiny{$\pm$0.011}} & 0.980{\tiny{$\pm$0.024}} & 0.987{\tiny{$\pm$0.012}} & \textbf{0.991{\tiny{$\pm$0.013}}} & 0.986{\tiny{$\pm$0.019}} & 0.973{\tiny{$\pm$0.049}} & 0.986{\tiny{$\pm$0.013}} \\
Pima & 0.756{\tiny{$\pm$0.026}} & 0.760{\tiny{$\pm$0.024}} & 0.767{\tiny{$\pm$0.023}} & \textbf{0.780{\tiny{$\pm$0.013}}} & 0.761{\tiny{$\pm$0.030}} & 0.757{\tiny{$\pm$0.032}} & 0.754{\tiny{$\pm$0.032}} \\
Vehicle2 & 0.967{\tiny{$\pm$0.015}} & 0.964{\tiny{$\pm$0.016}} & 0.962{\tiny{$\pm$0.020}} & 0.962{\tiny{$\pm$0.017}} & 0.965{\tiny{$\pm$0.009}} & 0.953{\tiny{$\pm$0.022}} & \textbf{0.970{\tiny{$\pm$0.010}}} \\
Vowel0 & \textbf{0.985{\tiny{$\pm$0.011}}} & \textbf{0.984{\tiny{$\pm$0.016}}} & 0.979{\tiny{$\pm$0.025}} & 0.975{\tiny{$\pm$0.022}} & 0.977{\tiny{$\pm$0.024}} & 0.959{\tiny{$\pm$0.035}} & 0.946{\tiny{$\pm$0.039}} \\
Wisconsin & 0.996{\tiny{$\pm$0.001}} & 0.996{\tiny{$\pm$0.004}} & 0.997{\tiny{$\pm$0.003}} & 0.997{\tiny{$\pm$0.003}} & \textbf{0.998{\tiny{$\pm$0.001}}} & 0.996{\tiny{$\pm$0.004}} & \textbf{0.998{\tiny{$\pm$0.001}}} \\
Yeast3 & 0.895{\tiny{$\pm$0.034}} & 0.891{\tiny{$\pm$0.054}} & 0.868{\tiny{$\pm$0.069}} & \textbf{0.915{\tiny{$\pm$0.030}}} & 0.891{\tiny{$\pm$0.029}} & 0.872{\tiny{$\pm$0.040}} & 0.857{\tiny{$\pm$0.059}} \\
\bottomrule
\end{tabular}%
}
\end{table}
content/discussion_and_conclusions/discussion_and_conclusions.tex
+33 -33
View File
@@ -3,7 +3,7 @@
\label{sec:discussion}
\addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{דיון}}
\subsection{Noise Robustness of \textsc{Roll}-AoC}
\subsection{Noise Robustness of \textsc{Roll}-AOC}
\label{sec:discussion:noise}
\addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{עמידות לרעש של ROLL-AOC}}
@@ -23,7 +23,7 @@
% strengthening the hypothesis that this is a property of the objective rather than an artifact
% of a specific dataset.
The data shows that \textsc{Roll}-AoC is a loss robust to label noise. In KEEL, we see that under the heaviest noise rate, \textsc{Roll}-AoC degrades least. In CiFAR10n, \textsc{Roll}-AoC suffers the least as label noise gets worse. Across other large datasets, \textsc{Roll}-AoC shows to be either competitive or leading. The experiments used a wide variety of data, both tabular and image-based, with varying levels of noise.
The data shows that \textsc{Roll}-AOC is a loss robust to label noise. In KEEL, we see that under the heaviest noise rate, \textsc{Roll}-AOC degrades least. In CIFAR-10N, \textsc{Roll}-AOC suffers the least as label noise gets worse. Across other large datasets, \textsc{Roll}-AOC shows to be either competitive or leading. The experiments used a wide variety of data, both tabular and image-based, with varying levels of noise.
Having the ability to optimize the desired metric (AUC) directly, rather than in a round about way, allows
our method to overcome challenging data. When using a per-sample loss, optimizing for a mislabeled sample
@@ -39,17 +39,17 @@ from NLP-extracted labels). The same advantage appearing across tabular data, na
images strengthens the case that noise robustness is a structural property of the distributional objective,
rather than an artifact of any particular dataset or noise type.
\subsection{\textsc{ROLL}-AoC Versus \textsc{GCE} and \textsc{MAE}}
\subsection{\textsc{ROLL}-AOC Versus \textsc{GCE} and \textsc{MAE}}
\label{sec:discussion:gce-mae}
\addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{ROLL-AOC מול GCE ו-MAE}}
\textsc{GCE} and \textsc{MAE} are used specifically for their robustness to noise, yet under 20\%
asymmetric noise on KEEL both degrade more than \textsc{Roll}-AoC (\textsc{GCE}: $-0.056$,
\textsc{MAE}: $-0.071$, \textsc{Roll}-AoC: $-0.033$). Whether this reflects a need for domain-specific
re-calibration or a fundamental limitation of per-sample objectives under label noise, \textsc{Roll}-AoC
asymmetric noise on KEEL both degrade more than \textsc{Roll}-AOC (\textsc{GCE}: $-0.056$,
\textsc{MAE}: $-0.071$, \textsc{Roll}-AOC: $-0.033$). Whether this reflects a need for domain-specific
re-calibration or a fundamental limitation of per-sample objectives under label noise, \textsc{Roll}-AOC
overcomes these challenges without any noise-specific adjustment.
\subsection{\textsc{Roll}-AoC Versus \textsc{LibAUC}: Stability Under Noise}
\subsection{\textsc{Roll}-AOC Versus \textsc{LibAUC}: Stability Under Noise}
\label{sec:discussion:libauc}
\addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{ROLL-AOC מול LibAUC: יציבות תחת רעש}}
@@ -66,26 +66,26 @@ overcomes these challenges without any noise-specific adjustment.
% Frame as a practical stability advantage: in noisy or heterogeneous settings,
% PESG-based optimization may be brittle in ways that a standard Adam + KDE loss is not.
While \textsc{Roll}-AoC shows improvement over per-sample losses on noisy datasets, it also shows
better performance than our LibAUC baseline, which also optimizes for the same metric directly.
While \textsc{Roll}-AOC shows improvement over per-sample losses on noisy datasets, it also outperforms
our LibAUC baseline, which also optimizes for AUC directly.
This improvement could be explained by \textsc{Roll}-AoC better tracking the real AUC value.
LibAUC's loss uses an imperfect surrogate loss. The AUC metric itself can only be acheived
exactly when using a stepwise function instead of the pairwise surrogate loss, which is impossible to use.
The surrogate losses used is the squared pairwise surrogate loss given:
This advantage can be explained by how closely each method's loss value tracks the actual AUC.
LibAUC replaces the intractable pairwise indicator $\mathbf{1}[f(x^+) > f(x^-)]$ with a squared surrogate:
\[
\ell(f(x^+), f(x^-)) = \bigl(1 - f(x^+) + f(x^-)\bigr)^2
\]
The squared surrogate and the indicator it replaces follow fundamentally different patterns: the squared
loss is sensitive to the magnitude of score differences, not only to their sign. As a result, the aggregate
loss value gives little information about the AUC actually achieved --- a model with large score margins
on easy pairs can have a low surrogate loss yet mediocre ranking quality on hard pairs.
$$\ell(f(x^+), f(x^-)) = \bigl(1 - f(x^+) + f(x^-)\bigr)^2$$
While this helps the math simplify in order to reduce the computation, the connection to the
actual AUC value is lost, and so the metric isn't directly optimized for.
\text{Roll}-AoC does not exactly estimate AUC either, yet the error is limited. As long as the bandwidth
is estimated properly, and the AUC is sampled at enough rates, one can expect a value very close to the real
AUC. These two pre-conditions are non-trivial to acheive, yet hold well enough for the experiments done as
part of this work.
This better tracking presents two benefits. Firstly, the gradients themselves are weighted in a more-exact way,
helping our model converge towards better performance. Secondly, it presents itself in best-epoch selection. Our training process takes the model with the lowest loss. Considering our loss tracks the real AUC better (to be exact, it tracks the area over the curve which complements AUC), this allows our best model selection to be more exact.
\textsc{Roll}-AOC does not track AUC exactly either, but its error is bounded. The loss is the numerical
integral of $1 - \operatorname{TPR}(\alpha)$ over 99 FPR grid points, directly approximating the area
over the ROC curve ($\mathrm{AOC} = 1 - \mathrm{AUC}$). As long as the KDE bandwidth is well-estimated
and 99 thresholds provide sufficient coverage, the loss value stays close to the true AOC.
The practical consequence is twofold: gradient weighting is more accurately aligned with ranking quality,
and best-epoch model selection --- which picks the epoch with the lowest loss --- selects the epoch with
the highest estimated AUC rather than the lowest surrogate value.
\subsection{Failure of the Single-Threshold Variant (\textsc{Roll}-TPR)}
\label{sec:discussion:roll-tpr}
@@ -118,7 +118,7 @@ of its score space, which exacerbated already existing overfitting issues and ma
to random initialization. On datasets such as Wisconsin and New-Thyroid1, this manifested as extreme
run-to-run variance ($\pm 0.441$ and $\pm 0.241$ AUC respectively under clean labels), where the
outcome was determined more by initialization than by the optimization itself.
This stands in direct contrast to \textsc{Roll}-AoC: as noted in \Cref{rem:aoc-coverage}, the 99
This stands in direct contrast to \textsc{Roll}-AOC: as noted in \Cref{rem:aoc-coverage}, the 99
simultaneously active thresholds spread gradient coverage across the full score range, making convergence
far more robust to initialization.
@@ -175,7 +175,7 @@ can perform poorly wherever the score distribution exhibits multiple scales of v
The number of FPR thresholds evaluated per forward pass is itself a trade-off. More thresholds yield a
finer approximation of the ROC curve and, as noted in \Cref{rem:aoc-coverage}, denser gradient coverage
across the score space --- both of which contribute to the robustness of \textsc{Roll}-AoC. Fewer
across the score space --- both of which contribute to the robustness of \textsc{Roll}-AOC. Fewer
thresholds reduce training cost proportionally but produce a coarser AUC estimate and sparser gradient
signal. Each threshold evaluation requires a KDE CDF query and a bisection inversion, each $O(n)$ in
the batch size, making the total per-batch cost $O(kn)$ where $k$ is the number of thresholds. In the
@@ -208,13 +208,13 @@ straightforward mitigation that has not yet been explored.
In this work, we present \textsc{ROLL} as a general framework for differentiable, ROC-based optimization
via parametric score distribution fitting. As the primary instantiation, we develop the KDE-based variant
and provide its full forward pass and gradient derivations. \textsc{ROLL}-AoC directly integrates the
and provide its full forward pass and gradient derivations. \textsc{ROLL}-AOC directly integrates the
estimated ROC curve over 99 evenly-spaced FPR thresholds --- a fundamentally different approach from
\textsc{LibAUC}'s pairwise formulation, which optimizes a squared surrogate over score pairs rather than
the curve itself.
We evaluated \textsc{ROLL}-AoC against losses designed for noise robustness and class imbalance. Across
datasets spanning tabular and image data with varying noise types and rates, we show that \textsc{ROLL}-AoC
We evaluated \textsc{ROLL}-AOC against losses designed for noise robustness and class imbalance. Across
datasets spanning tabular and image data with varying noise types and rates, we show that \textsc{ROLL}-AOC
handles these challenges competitively, often leading, despite noise robustness not being an explicit
design objective.
@@ -222,9 +222,9 @@ design objective.
\label{sec:conclusions:limitations}
\addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{מגבלות}}
\textsc{Roll}-AoC has several practical limitations worth noting.
\textsc{Roll}-AOC has several practical limitations worth noting.
The KDE instantiation uses a single bandwidth per class, which cannot accurately represent multimodal or heavy-tailed score distributions.
Computational cost is significant: evaluating 99 FPR thresholds per forward pass makes \textsc{Roll}-AoC considerably more expensive per training step than per-sample losses or \textsc{LibAUC}'s pairwise formulation.
Computational cost is significant: evaluating 99 FPR thresholds per forward pass makes \textsc{Roll}-AOC considerably more expensive per training step than per-sample losses or \textsc{LibAUC}'s pairwise formulation.
The single-threshold variant (\textsc{Roll}-TPR) is brittle in practice; gradient locality concentrates updates near one operating point and the benefit rarely transfers to held-out data.
Finally, the framework currently lacks formal convergence guarantees; the empirical results are encouraging but not theoretically grounded.
@@ -242,4 +242,4 @@ Variable-bandwidth (adaptive) KDE methods address the single-bandwidth limitatio
Formal convergence guarantees for KDE-ROLL under standard assumptions (batch size, bandwidth decay schedule, smoothness) remain an open problem, and would also clarify how the bandwidth scheduler interacts with gradient variance across training.
\paragraph{Partial and weighted AUC.}
Restricting or reweighting the FPR grid to emphasize specific operating regions is a straightforward modification to \textsc{Roll}-AoC, and a natural fit for settings such as medical imaging where only a narrow FPR range is clinically actionable.
Restricting or reweighting the FPR grid to emphasize specific operating regions is a straightforward modification to \textsc{Roll}-AOC, and a natural fit for settings such as medical imaging where only a narrow FPR range is clinically actionable.
content/experiments/experiments.tex
+58 -155
View File
@@ -31,15 +31,15 @@ Yeast3 & 1484 & 163 &1321 & 8.1 & Yeast protein localisation (UCI) \\
\end{table}
\paragraph{CIFAR-10N.}
CIFAR-10 is a widely used image dataset of 60{,}000 32x32 color images belonging to 10 mutuall exclusive classes. CIFAR-10N~\cite{wei2022cifar10n} augments the 50{,}000 training set labels with real human-annotated noisy labels collected from Amazon Mechanical Turk~\cite{wei2022cifar10n}. We form a binary task (class~1, automobile, vs.\ rest; IR~$\approx 9$:1) and train under three noise conditions:
CIFAR-10 is a widely used image dataset of 60{,}000 32x32 color images belonging to 10 mutually exclusive classes. CIFAR-10N~\cite{wei2022cifar10n} augments the 50{,}000 training set labels with real human-annotated noisy labels collected from Amazon Mechanical Turk~\cite{wei2022cifar10n}. We form a binary task (class~1, automobile, vs.\ rest; IR~$\approx 9$:1) and train under three noise conditions:
\emph{clean} (original CIFAR-10 labels, 0\% noise),
\emph{aggre} (majority-vote aggregate of crowdworker annotations, $\approx$9\% noise), and
\emph{worse} (single worst-annotator labels, $\approx$40\% noise).
This allows us to measure the degredation of the model's performance under increasing label noise conditions.
This allows us to measure the degradation of the model's performance under increasing label noise conditions.
The architecture tested is a three-block convolutional network trained from scratch.
\paragraph{Large tabular datasets.}
We include three large-scale tabular datasets spanning diverse imbalanced real-world settings.
\paragraph{Large datasets.}
We include four large-scale datasets spanning diverse imbalanced real-world settings.
\begin{itemize}
\item \textbf{Bank Marketing}~\cite{moro2014bank}, UCI repository; Portuguese bank telephone marketing campaigns (Moro et al., 2014). $n = 41{,}188$ samples, 48 features after one-hot encoding. Positive class: client subscribed to term deposit. IR~$\approx 8$:1 ($\approx$11.3\% positive). No label noise.
@@ -47,13 +47,10 @@ We include three large-scale tabular datasets spanning diverse imbalanced real-w
\item \textbf{Credit Card Fraud}~\cite{dalpozzolo2015calibrating}, Kaggle / ULB (Worldline + MLG, 2013). $n = 284{,}807$ transactions, 30 features (V1--V28 are PCA-transformed; Time and Amount rescaled). Extreme imbalance: 492 confirmed frauds, IR~$\approx 577$:1. Labels reflect confirmed-fraud reports; undetected fraudulent transactions constitute a structural source of class-conditional false negatives~\cite{tittelfitz2023fpr}.
\item \textbf{Home Credit}~\cite{homecredit2018kaggle}, Kaggle Home Credit Default Risk competition (2018). $n = 307{,}511$ loan applications, $\approx$120 numerical features from the primary application table. Positive class: loan default. IR~$\approx 11.4$:1 ($\approx$8\% default rate). Labels are proxy labels (repayment outcome); no injected noise.
\item \textbf{NIH ChestX-ray14}~\cite{wang2017chestxray14}, a large-scale medical chest radiograph dataset of $112{,}120$ X-rays from $30{,}805$ patients annotated with 14 pathology labels. Labels were extracted from radiological reports via NLP; the dataset authors estimate accuracy at $>$90\%~\cite{wang2017chestxray14}, though subsequent audits have found the true error rate substantially higher~\cite{oakdenrayner2020exploring}, making this a naturally noisy benchmark. We form a binary task targeting \emph{pleural effusion} detection (Effusion vs.\ rest; IR~$\approx 7.5$:1). Unlike the other three datasets, inputs here are not raw tabular features: we extract $1{,}024$-dimensional latent representations using a \mbox{DenseNet-121} pretrained on the dataset via the TorchXRayVision library~\cite{cohen2022torchxrayvision}, with global average pooling applied to the penultimate feature map, and pass these frozen embeddings to a lightweight $1024{\to}512{\to}128{\to}1$ MLP with dropout. This transfer-learning setup lets us evaluate ROLL without GPU-intensive end-to-end training.
\end{itemize}
\paragraph{NIH ChestX-ray14.}
NIH ChestX-ray14~\cite{wang2017chestxray14} is a large-scale medical chest radiograph dataset of $112{,}120$ X-rays from $30{,}805$ patients, annotated with 14 unique labels. Labels were extracted from radiological reports via natural language processing; the dataset authors estimate labelling accuracy at $>$90\%~\cite{wang2017chestxray14}, though subsequent audits have found the true error rate substantially higher, with positive predictive values 10--30 percentage points below the reported figures~\cite{oakdenrayner2020exploring}, making this a naturally noisy benchmark. We form a binary task targeting \emph{pleural effusion} detection (Effusion vs.\ rest; IR~$\approx 7.5$:1).
We extract $1{,}024$-dimensional latent representations using a \mbox{DenseNet-121} pretrained on the dataset via the TorchXRayVision library~\cite{cohen2022torchxrayvision}, with global average pooling applied to the penultimate feature map. These frozen embeddings are passed to a lightweight $1024{\to}512{\to}128{\to}1$ MLP with dropout, treating image representations as tabular features. This setup evaluates ROLL in a transfer-learning scenario and avoids GPU-intensive end-to-end training.
\section{Performance Measures}\label{sec:experiments:performance_measures}
\addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{מדדי ביצועים}}
@@ -63,7 +60,7 @@ We report the \emph{area under the ROC curve} (AUC) averaged over independent tr
\label{sec:experiments:configurations}
\addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{קונפיגורציה ופרמטרים}}
We compare \textsc{Roll}-AoC (the primary proposed loss, optimising the area under the ROC curve via KDE) against the following baselines:
We compare \textsc{Roll}-AOC (the primary proposed loss, optimising the area under the ROC curve via KDE) against the following baselines:
\begin{itemize}
\item \textbf{BCE-W}: binary cross-entropy with class-frequency inverse weighting.
@@ -79,102 +76,45 @@ All methods use the same network architecture, optimiser (Adam for non-LibAUC me
\section{Results}\label{sec:experiments:results}
\addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{תוצאות}}
% TODO: Review which KEEL noise tables to keep in the main body.
% Low noise levels (5%, 10%) show little differentiation and may not add much to the narrative.
% Consider: keep only 20% noise table here (clearest signal, Roll-AoC leads on 6/12 datasets)
% and move 5%/10% tables to an appendix, referencing them from here for completeness.
\subsection{KEEL Benchmark}\label{sec:results:keel}
\Cref{tab:results:keel} reports mean test AUC across the 12 KEEL datasets under clean labels.
No single method dominates: on easy datasets (Iris0, Wisconsin, Vowel0, Vehicle2, Glass6) nearly all methods converge to similar AUCs, leaving little room for differentiation.
On the harder, more imbalanced datasets the picture is more varied.
\textsc{Roll}-AoC competes well but does not claim the top position on most individual datasets, establishing it as a competitive approach.
\textsc{Roll}-AOC competes well but does not claim the top position on most individual datasets, establishing it as a competitive approach.
To assess label-noise robustness, we inject symmetric label noise. A fraction of all class labels are randomly flipped.
We evaluate at noise levels of $5\%, 10\%, 20\%$, running 10 independent episodes per dataset per noise level.
\Cref{tab:results:keel-noise-05,tab:results:keel-noise-10,tab:results:keel-noise-20} show per-dataset results at each noise level.
The randomly flipped labels change each episode, yet are consistent between configurations, ensuring every configuration, per episode, sees the same labels, changed in the same way.
Full per-dataset results at 5\% and 10\% noise are given in \Cref{appendix:keel-noise} (\Cref{tab:results:keel-noise-05,tab:results:keel-noise-10}); the 20\% results, where differentiation is strongest, are shown in \Cref{tab:results:keel-noise-20} below.
At $0\%$ noise, results are quite similar across configurations. \textsc{Roll}-AOC trails right behind the leader in most configurations. Since datasets are small, the \textbf{std} given for each point is relatively high, and quite a bit of variability is expected. Still, we see that, with few exceptions, results between configurations seem within throwing distance of eachother.
As noise increases the ranking shifts markedly: at $20\%$ noise \textsc{Roll}-AoC reaches the top on average (0.847), while traditionally noise-robust losses --- \textsc{GCE} (0.831) and \textsc{MAE} (0.804) --- degrade substantially.
The per-dataset tables reveal that this trend is consistent across most datasets rather than driven by a small subset: at $20\%$ noise \textsc{Roll}-AoC leads on 6 of 12 datasets (Glass0, Glass1, Glass2, Pima, Vehicle2, Yeast3), while no other method leads on more than 3.
From $0\%$ to $20\%$ noise, \textsc{Roll}-AoC drops 0.033 in mean AUC (3.7\%), compared to 0.056 for \textsc{GCE} (6.3\%), 0.071 for \textsc{MAE} (8.1\%), and 0.073 for \textsc{LibAUC} (8.3\%).
At $0\%$ noise, results are quite similar across configurations. \textsc{Roll}-AOC trails right behind the leader in most configurations. Since datasets are small, the \textbf{std} given for each point is relatively high, and quite a bit of variability is expected. Still, with few exceptions, results between configurations are within throwing distance of each other.
As noise increases the ranking shifts markedly: at $20\%$ noise \textsc{Roll}-AOC reaches the top on average (0.847), while traditionally noise-robust losses --- \textsc{GCE} (0.831) and \textsc{MAE} (0.804) --- degrade substantially.
The per-dataset results in \Cref{tab:results:keel-noise-20} reveal that this trend is consistent across most datasets rather than driven by a small subset: \textsc{Roll}-AOC leads on 6 of 12 datasets (Glass0, Glass1, Glass2, Pima, Vehicle2, Yeast3), while no other method leads on more than 3.
From $0\%$ to $20\%$ noise, \textsc{Roll}-AOC drops 0.033 in mean AUC (3.7\%), compared to 0.056 for \textsc{GCE} (6.3\%), 0.071 for \textsc{MAE} (8.1\%), and 0.073 for \textsc{LibAUC} (8.3\%).
This supports the hypothesis that ROLL's distributional objective provides implicit noise robustness: mislabelled samples shift the fitted score distribution gradually rather than producing isolated large-gradient outliers.
% TODO: audit bold/emphasis across all result tables — check that bold marks the correct per-row best and that no ties or near-ties are missed or incorrectly marked.
\begin{table}[htbp]
\centering
\caption{Mean test AUC ($\pm$\,std, $n=5$ episodes) on KEEL benchmark datasets under clean labels. Best per row in \textbf{bold}.}
\label{tab:results:keel}
\resizebox{\textwidth}{!}{%
\begin{tabular}{lcccccccc}
\begin{tabular}{lccccccc}
\toprule
Dataset & \textsc{Roll}-AoC & \textsc{Roll}-TPR & BCE-W & Focal & Asym. & GCE & LibAUC & MAE \\
Dataset & \textsc{Roll}-AOC & BCE-W & Focal & Asym. & GCE & LibAUC & MAE \\
\midrule
Glass0 & 0.796{\tiny{$\pm$0.046}} & 0.826{\tiny{$\pm$0.032}} & 0.839{\tiny{$\pm$0.033}} & 0.802{\tiny{$\pm$0.020}} & 0.814{\tiny{$\pm$0.034}} & 0.818{\tiny{$\pm$0.032}} & \textbf{0.843{\tiny{$\pm$0.035}}} & 0.826{\tiny{$\pm$0.037}} \\
Glass1 & 0.722{\tiny{$\pm$0.033}} & 0.698{\tiny{$\pm$0.042}} & 0.710{\tiny{$\pm$0.018}} & 0.712{\tiny{$\pm$0.036}} & 0.744{\tiny{$\pm$0.044}} & 0.742{\tiny{$\pm$0.039}} & 0.723{\tiny{$\pm$0.054}} & \textbf{0.769{\tiny{$\pm$0.031}}} \\
Glass2 & 0.723{\tiny{$\pm$0.062}} & 0.701{\tiny{$\pm$0.055}} & 0.661{\tiny{$\pm$0.069}} & 0.677{\tiny{$\pm$0.072}} & \textbf{0.750{\tiny{$\pm$0.075}}} & 0.689{\tiny{$\pm$0.023}} & 0.719{\tiny{$\pm$0.084}} & 0.679{\tiny{$\pm$0.018}} \\
Glass6 & 0.975{\tiny{$\pm$0.014}} & 0.975{\tiny{$\pm$0.027}} & 0.972{\tiny{$\pm$0.025}} & 0.974{\tiny{$\pm$0.020}} & 0.982{\tiny{$\pm$0.013}} & 0.971{\tiny{$\pm$0.020}} & 0.969{\tiny{$\pm$0.012}} & \textbf{0.985{\tiny{$\pm$0.006}}} \\
Haberman & 0.696{\tiny{$\pm$0.045}} & 0.675{\tiny{$\pm$0.063}} & 0.736{\tiny{$\pm$0.012}} & \textbf{0.742{\tiny{$\pm$0.025}}} & 0.741{\tiny{$\pm$0.018}} & 0.737{\tiny{$\pm$0.016}} & 0.616{\tiny{$\pm$0.116}} & 0.671{\tiny{$\pm$0.102}} \\
Iris0 & \textbf{1.000{\tiny{$\pm$0.000}}} & 0.981{\tiny{$\pm$0.027}} & 0.900{\tiny{$\pm$0.224}} & \textbf{1.000{\tiny{$\pm$0.000}}} & \textbf{1.000{\tiny{$\pm$0.000}}} & \textbf{1.000{\tiny{$\pm$0.000}}} & \textbf{1.000{\tiny{$\pm$0.000}}} & 0.900{\tiny{$\pm$0.224}} \\
New-Thyroid1 & 0.989{\tiny{$\pm$0.010}} & 0.845{\tiny{$\pm$0.241}} & 0.992{\tiny{$\pm$0.005}} & 0.977{\tiny{$\pm$0.033}} & 0.988{\tiny{$\pm$0.007}} & 0.987{\tiny{$\pm$0.011}} & 0.985{\tiny{$\pm$0.012}} & \textbf{0.994{\tiny{$\pm$0.003}}} \\
Pima & 0.778{\tiny{$\pm$0.024}} & 0.755{\tiny{$\pm$0.014}} & \textbf{0.787{\tiny{$\pm$0.014}}} & 0.775{\tiny{$\pm$0.020}} & 0.771{\tiny{$\pm$0.019}} & 0.785{\tiny{$\pm$0.005}} & 0.761{\tiny{$\pm$0.015}} & 0.778{\tiny{$\pm$0.005}} \\
Vehicle2 & 0.982{\tiny{$\pm$0.003}} & 0.966{\tiny{$\pm$0.020}} & \textbf{0.984{\tiny{$\pm$0.004}}} & 0.982{\tiny{$\pm$0.003}} & 0.980{\tiny{$\pm$0.004}} & 0.982{\tiny{$\pm$0.004}} & 0.982{\tiny{$\pm$0.003}} & 0.983{\tiny{$\pm$0.004}} \\
Vowel0 & 0.993{\tiny{$\pm$0.005}} & 0.987{\tiny{$\pm$0.002}} & 0.995{\tiny{$\pm$0.004}} & 0.996{\tiny{$\pm$0.003}} & 0.995{\tiny{$\pm$0.003}} & 0.997{\tiny{$\pm$0.003}} & 0.991{\tiny{$\pm$0.007}} & \textbf{0.998{\tiny{$\pm$0.001}}} \\
Wisconsin & 0.997{\tiny{$\pm$0.002}} & 0.800{\tiny{$\pm$0.441}} & \textbf{0.999{\tiny{$\pm$0.000}}} & \textbf{0.999{\tiny{$\pm$0.001}}} & 0.999{\tiny{$\pm$0.001}} & 0.999{\tiny{$\pm$0.001}} & 0.999{\tiny{$\pm$0.000}} & 0.998{\tiny{$\pm$0.001}} \\
Yeast3 & 0.906{\tiny{$\pm$0.024}} & 0.908{\tiny{$\pm$0.021}} & 0.894{\tiny{$\pm$0.008}} & 0.908{\tiny{$\pm$0.019}} & \textbf{0.935{\tiny{$\pm$0.014}}} & 0.927{\tiny{$\pm$0.006}} & \textbf{0.935{\tiny{$\pm$0.016}}} & 0.915{\tiny{$\pm$0.010}} \\
\bottomrule
\end{tabular}%
}
\end{table}
\begin{table}[htbp]
\centering
\caption{Mean test AUC ($\pm$\,std, $n=10$ episodes) on KEEL datasets at $\epsilon=5\%$ asymmetric label noise. Best per row in \textbf{bold}.}
\label{tab:results:keel-noise-05}
\resizebox{\textwidth}{!}{%
\begin{tabular}{lcccccccc}
\toprule
Dataset & \textsc{Roll}-AoC & \textsc{Roll}-TPR & BCE-W & Focal & Asym. & GCE & LibAUC & MAE \\
\midrule
Glass0 & 0.797{\tiny{$\pm$0.046}} & 0.759{\tiny{$\pm$0.060}} & 0.786{\tiny{$\pm$0.019}} & \textbf{0.806{\tiny{$\pm$0.030}}} & 0.792{\tiny{$\pm$0.041}} & 0.794{\tiny{$\pm$0.029}} & 0.791{\tiny{$\pm$0.056}} & 0.792{\tiny{$\pm$0.058}} \\
Glass1 & 0.728{\tiny{$\pm$0.037}} & 0.702{\tiny{$\pm$0.041}} & 0.736{\tiny{$\pm$0.032}} & 0.730{\tiny{$\pm$0.074}} & 0.704{\tiny{$\pm$0.028}} & 0.717{\tiny{$\pm$0.059}} & 0.694{\tiny{$\pm$0.060}} & \textbf{0.738{\tiny{$\pm$0.047}}} \\
Glass2 & 0.677{\tiny{$\pm$0.095}} & 0.607{\tiny{$\pm$0.114}} & 0.665{\tiny{$\pm$0.128}} & 0.581{\tiny{$\pm$0.091}} & 0.662{\tiny{$\pm$0.050}} & 0.651{\tiny{$\pm$0.078}} & 0.614{\tiny{$\pm$0.085}} & \textbf{0.686{\tiny{$\pm$0.091}}} \\
Glass6 & 0.972{\tiny{$\pm$0.021}} & 0.909{\tiny{$\pm$0.068}} & 0.972{\tiny{$\pm$0.035}} & 0.978{\tiny{$\pm$0.016}} & \textbf{0.981{\tiny{$\pm$0.016}}} & 0.947{\tiny{$\pm$0.049}} & 0.967{\tiny{$\pm$0.036}} & 0.971{\tiny{$\pm$0.031}} \\
Haberman & 0.666{\tiny{$\pm$0.094}} & 0.562{\tiny{$\pm$0.148}} & 0.627{\tiny{$\pm$0.124}} & 0.694{\tiny{$\pm$0.111}} & 0.693{\tiny{$\pm$0.065}} & 0.687{\tiny{$\pm$0.099}} & 0.651{\tiny{$\pm$0.125}} & \textbf{0.716{\tiny{$\pm$0.055}}} \\
Iris0 & 0.990{\tiny{$\pm$0.014}} & 0.770{\tiny{$\pm$0.284}} & 0.998{\tiny{$\pm$0.006}} & 0.999{\tiny{$\pm$0.002}} & 0.999{\tiny{$\pm$0.002}} & \textbf{1.000{\tiny{$\pm$0.000}}} & \textbf{1.000{\tiny{$\pm$0.000}}} & 0.999{\tiny{$\pm$0.003}} \\
New-Thyroid1 & 0.984{\tiny{$\pm$0.010}} & 0.609{\tiny{$\pm$0.408}} & 0.986{\tiny{$\pm$0.007}} & 0.971{\tiny{$\pm$0.046}} & 0.989{\tiny{$\pm$0.006}} & 0.987{\tiny{$\pm$0.019}} & 0.983{\tiny{$\pm$0.023}} & \textbf{0.991{\tiny{$\pm$0.012}}} \\
Pima & 0.780{\tiny{$\pm$0.021}} & 0.761{\tiny{$\pm$0.030}} & 0.784{\tiny{$\pm$0.016}} & 0.780{\tiny{$\pm$0.019}} & 0.767{\tiny{$\pm$0.023}} & \textbf{0.787{\tiny{$\pm$0.011}}} & 0.756{\tiny{$\pm$0.023}} & 0.775{\tiny{$\pm$0.023}} \\
Vehicle2 & 0.975{\tiny{$\pm$0.007}} & 0.954{\tiny{$\pm$0.043}} & \textbf{0.977{\tiny{$\pm$0.006}}} & 0.974{\tiny{$\pm$0.010}} & 0.969{\tiny{$\pm$0.016}} & 0.976{\tiny{$\pm$0.007}} & 0.975{\tiny{$\pm$0.008}} & 0.976{\tiny{$\pm$0.009}} \\
Vowel0 & 0.988{\tiny{$\pm$0.012}} & 0.959{\tiny{$\pm$0.028}} & 0.981{\tiny{$\pm$0.019}} & 0.989{\tiny{$\pm$0.017}} & \textbf{0.991{\tiny{$\pm$0.006}}} & 0.985{\tiny{$\pm$0.014}} & 0.973{\tiny{$\pm$0.025}} & 0.978{\tiny{$\pm$0.014}} \\
Wisconsin & 0.998{\tiny{$\pm$0.002}} & 0.997{\tiny{$\pm$0.004}} & 0.998{\tiny{$\pm$0.001}} & \textbf{0.999{\tiny{$\pm$0.001}}} & \textbf{0.999{\tiny{$\pm$0.001}}} & 0.998{\tiny{$\pm$0.001}} & 0.998{\tiny{$\pm$0.001}} & 0.998{\tiny{$\pm$0.001}} \\
Yeast3 & 0.900{\tiny{$\pm$0.020}} & 0.843{\tiny{$\pm$0.070}} & 0.914{\tiny{$\pm$0.028}} & 0.915{\tiny{$\pm$0.034}} & \textbf{0.916{\tiny{$\pm$0.026}}} & \textbf{0.916{\tiny{$\pm$0.032}}} & 0.908{\tiny{$\pm$0.029}} & 0.909{\tiny{$\pm$0.024}} \\
\bottomrule
\end{tabular}%
}
\end{table}
\begin{table}[htbp]
\centering
\caption{Mean test AUC ($\pm$\,std, $n=10$ episodes) on KEEL datasets at $\epsilon=10\%$ asymmetric label noise. Best per row in \textbf{bold}.}
\label{tab:results:keel-noise-10}
\resizebox{\textwidth}{!}{%
\begin{tabular}{lcccccccc}
\toprule
Dataset & \textsc{Roll}-AoC & \textsc{Roll}-TPR & BCE-W & Focal & Asym. & GCE & LibAUC & MAE \\
\midrule
Glass0 & 0.773{\tiny{$\pm$0.070}} & 0.767{\tiny{$\pm$0.046}} & 0.789{\tiny{$\pm$0.037}} & 0.771{\tiny{$\pm$0.037}} & \textbf{0.821{\tiny{$\pm$0.026}}} & 0.786{\tiny{$\pm$0.057}} & 0.800{\tiny{$\pm$0.032}} & 0.807{\tiny{$\pm$0.050}} \\
Glass1 & 0.719{\tiny{$\pm$0.040}} & 0.725{\tiny{$\pm$0.029}} & 0.716{\tiny{$\pm$0.079}} & 0.718{\tiny{$\pm$0.056}} & 0.710{\tiny{$\pm$0.049}} & 0.729{\tiny{$\pm$0.048}} & 0.675{\tiny{$\pm$0.070}} & \textbf{0.738{\tiny{$\pm$0.060}}} \\
Glass2 & 0.695{\tiny{$\pm$0.075}} & 0.649{\tiny{$\pm$0.119}} & 0.644{\tiny{$\pm$0.072}} & 0.687{\tiny{$\pm$0.127}} & 0.655{\tiny{$\pm$0.048}} & \textbf{0.713{\tiny{$\pm$0.107}}} & 0.637{\tiny{$\pm$0.093}} & 0.673{\tiny{$\pm$0.111}} \\
Glass6 & 0.947{\tiny{$\pm$0.050}} & 0.910{\tiny{$\pm$0.067}} & \textbf{0.973{\tiny{$\pm$0.026}}} & 0.950{\tiny{$\pm$0.041}} & 0.971{\tiny{$\pm$0.024}} & 0.938{\tiny{$\pm$0.047}} & 0.937{\tiny{$\pm$0.048}} & 0.935{\tiny{$\pm$0.054}} \\
Haberman & 0.638{\tiny{$\pm$0.098}} & 0.564{\tiny{$\pm$0.144}} & 0.700{\tiny{$\pm$0.074}} & 0.652{\tiny{$\pm$0.106}} & 0.658{\tiny{$\pm$0.087}} & 0.636{\tiny{$\pm$0.112}} & 0.694{\tiny{$\pm$0.129}} & \textbf{0.732{\tiny{$\pm$0.022}}} \\
Iris0 & 0.991{\tiny{$\pm$0.020}} & 0.879{\tiny{$\pm$0.209}} & 0.950{\tiny{$\pm$0.158}} & 0.950{\tiny{$\pm$0.158}} & \textbf{1.000{\tiny{$\pm$0.000}}} & \textbf{1.000{\tiny{$\pm$0.001}}} & \textbf{1.000{\tiny{$\pm$0.000}}} & \textbf{1.000{\tiny{$\pm$0.000}}} \\
New-Thyroid1 & 0.986{\tiny{$\pm$0.011}} & 0.588{\tiny{$\pm$0.358}} & 0.980{\tiny{$\pm$0.024}} & 0.987{\tiny{$\pm$0.012}} & \textbf{0.991{\tiny{$\pm$0.013}}} & 0.986{\tiny{$\pm$0.019}} & 0.973{\tiny{$\pm$0.049}} & 0.986{\tiny{$\pm$0.013}} \\
Pima & 0.756{\tiny{$\pm$0.026}} & 0.752{\tiny{$\pm$0.027}} & 0.760{\tiny{$\pm$0.024}} & 0.767{\tiny{$\pm$0.023}} & \textbf{0.780{\tiny{$\pm$0.013}}} & 0.761{\tiny{$\pm$0.030}} & 0.757{\tiny{$\pm$0.032}} & 0.754{\tiny{$\pm$0.032}} \\
Vehicle2 & 0.967{\tiny{$\pm$0.015}} & 0.939{\tiny{$\pm$0.020}} & 0.964{\tiny{$\pm$0.016}} & 0.962{\tiny{$\pm$0.020}} & 0.962{\tiny{$\pm$0.017}} & 0.965{\tiny{$\pm$0.009}} & 0.953{\tiny{$\pm$0.022}} & \textbf{0.970{\tiny{$\pm$0.010}}} \\
Vowel0 & \textbf{0.985{\tiny{$\pm$0.011}}} & 0.946{\tiny{$\pm$0.036}} & \textbf{0.984{\tiny{$\pm$0.016}}} & 0.979{\tiny{$\pm$0.025}} & 0.975{\tiny{$\pm$0.022}} & 0.977{\tiny{$\pm$0.024}} & 0.959{\tiny{$\pm$0.035}} & 0.946{\tiny{$\pm$0.039}} \\
Wisconsin & 0.996{\tiny{$\pm$0.001}} & 0.717{\tiny{$\pm$0.453}} & 0.996{\tiny{$\pm$0.004}} & 0.997{\tiny{$\pm$0.003}} & 0.997{\tiny{$\pm$0.003}} & \textbf{0.998{\tiny{$\pm$0.001}}} & 0.996{\tiny{$\pm$0.004}} & \textbf{0.998{\tiny{$\pm$0.001}}} \\
Yeast3 & 0.895{\tiny{$\pm$0.034}} & 0.810{\tiny{$\pm$0.097}} & 0.891{\tiny{$\pm$0.054}} & 0.868{\tiny{$\pm$0.069}} & \textbf{0.915{\tiny{$\pm$0.030}}} & 0.891{\tiny{$\pm$0.029}} & 0.872{\tiny{$\pm$0.040}} & 0.857{\tiny{$\pm$0.059}} \\
Glass0 & 0.796{\tiny{$\pm$0.046}} & 0.839{\tiny{$\pm$0.033}} & 0.802{\tiny{$\pm$0.020}} & 0.814{\tiny{$\pm$0.034}} & 0.818{\tiny{$\pm$0.032}} & \textbf{0.843{\tiny{$\pm$0.035}}} & 0.826{\tiny{$\pm$0.037}} \\
Glass1 & 0.722{\tiny{$\pm$0.033}} & 0.710{\tiny{$\pm$0.018}} & 0.712{\tiny{$\pm$0.036}} & 0.744{\tiny{$\pm$0.044}} & 0.742{\tiny{$\pm$0.039}} & 0.723{\tiny{$\pm$0.054}} & \textbf{0.769{\tiny{$\pm$0.031}}} \\
Glass2 & 0.723{\tiny{$\pm$0.062}} & 0.661{\tiny{$\pm$0.069}} & 0.677{\tiny{$\pm$0.072}} & \textbf{0.750{\tiny{$\pm$0.075}}} & 0.689{\tiny{$\pm$0.023}} & 0.719{\tiny{$\pm$0.084}} & 0.679{\tiny{$\pm$0.018}} \\
Glass6 & 0.975{\tiny{$\pm$0.014}} & 0.972{\tiny{$\pm$0.025}} & 0.974{\tiny{$\pm$0.020}} & 0.982{\tiny{$\pm$0.013}} & 0.971{\tiny{$\pm$0.020}} & 0.969{\tiny{$\pm$0.012}} & \textbf{0.985{\tiny{$\pm$0.006}}} \\
Haberman & 0.696{\tiny{$\pm$0.045}} & 0.736{\tiny{$\pm$0.012}} & \textbf{0.742{\tiny{$\pm$0.025}}} & 0.741{\tiny{$\pm$0.018}} & 0.737{\tiny{$\pm$0.016}} & 0.616{\tiny{$\pm$0.116}} & 0.671{\tiny{$\pm$0.102}} \\
Iris0 & \textbf{1.000{\tiny{$\pm$0.000}}} & 0.900{\tiny{$\pm$0.224}} & \textbf{1.000{\tiny{$\pm$0.000}}} & \textbf{1.000{\tiny{$\pm$0.000}}} & \textbf{1.000{\tiny{$\pm$0.000}}} & \textbf{1.000{\tiny{$\pm$0.000}}} & 0.900{\tiny{$\pm$0.224}} \\
New-Thyroid1 & 0.989{\tiny{$\pm$0.010}} & 0.992{\tiny{$\pm$0.005}} & 0.977{\tiny{$\pm$0.033}} & 0.988{\tiny{$\pm$0.007}} & 0.987{\tiny{$\pm$0.011}} & 0.985{\tiny{$\pm$0.012}} & \textbf{0.994{\tiny{$\pm$0.003}}} \\
Pima & 0.778{\tiny{$\pm$0.024}} & \textbf{0.787{\tiny{$\pm$0.014}}} & 0.775{\tiny{$\pm$0.020}} & 0.771{\tiny{$\pm$0.019}} & 0.785{\tiny{$\pm$0.005}} & 0.761{\tiny{$\pm$0.015}} & 0.778{\tiny{$\pm$0.005}} \\
Vehicle2 & 0.982{\tiny{$\pm$0.003}} & \textbf{0.984{\tiny{$\pm$0.004}}} & 0.982{\tiny{$\pm$0.003}} & 0.980{\tiny{$\pm$0.004}} & 0.982{\tiny{$\pm$0.004}} & 0.982{\tiny{$\pm$0.003}} & 0.983{\tiny{$\pm$0.004}} \\
Vowel0 & 0.993{\tiny{$\pm$0.005}} & 0.995{\tiny{$\pm$0.004}} & 0.996{\tiny{$\pm$0.003}} & 0.995{\tiny{$\pm$0.003}} & 0.997{\tiny{$\pm$0.003}} & 0.991{\tiny{$\pm$0.007}} & \textbf{0.998{\tiny{$\pm$0.001}}} \\
Wisconsin & 0.997{\tiny{$\pm$0.002}} & \textbf{0.999{\tiny{$\pm$0.000}}} & \textbf{0.999{\tiny{$\pm$0.001}}} & 0.999{\tiny{$\pm$0.001}} & 0.999{\tiny{$\pm$0.001}} & 0.999{\tiny{$\pm$0.000}} & 0.998{\tiny{$\pm$0.001}} \\
Yeast3 & 0.906{\tiny{$\pm$0.024}} & 0.894{\tiny{$\pm$0.008}} & 0.908{\tiny{$\pm$0.019}} & \textbf{0.935{\tiny{$\pm$0.014}}} & 0.927{\tiny{$\pm$0.006}} & \textbf{0.935{\tiny{$\pm$0.016}}} & 0.915{\tiny{$\pm$0.010}} \\
\bottomrule
\end{tabular}%
}
@@ -185,22 +125,22 @@ Yeast3 & 0.895{\tiny{$\pm$0.034}} & 0.810{\tiny{$\pm$0.097}} & 0.891{\tiny
\caption{Mean test AUC ($\pm$\,std, $n=10$ episodes) on KEEL datasets at $\epsilon=20\%$ asymmetric label noise. Best per row in \textbf{bold}.}
\label{tab:results:keel-noise-20}
\resizebox{\textwidth}{!}{%
\begin{tabular}{lcccccccc}
\begin{tabular}{lccccccc}
\toprule
Dataset & \textsc{Roll}-AoC & \textsc{Roll}-TPR & BCE-W & Focal & Asym. & GCE & LibAUC & MAE \\
Dataset & \textsc{Roll}-AOC & BCE-W & Focal & Asym. & GCE & LibAUC & MAE \\
\midrule
Glass0 & \textbf{0.801{\tiny{$\pm$0.042}}} & 0.758{\tiny{$\pm$0.049}} & 0.767{\tiny{$\pm$0.042}} & 0.796{\tiny{$\pm$0.040}} & 0.793{\tiny{$\pm$0.051}} & 0.765{\tiny{$\pm$0.052}} & 0.745{\tiny{$\pm$0.089}} & 0.791{\tiny{$\pm$0.044}} \\
Glass1 & \textbf{0.725{\tiny{$\pm$0.036}}} & 0.682{\tiny{$\pm$0.068}} & 0.718{\tiny{$\pm$0.054}} & 0.714{\tiny{$\pm$0.036}} & 0.700{\tiny{$\pm$0.039}} & 0.706{\tiny{$\pm$0.041}} & 0.635{\tiny{$\pm$0.052}} & 0.689{\tiny{$\pm$0.060}} \\
Glass2 & \textbf{0.622{\tiny{$\pm$0.110}}} & 0.585{\tiny{$\pm$0.131}} & 0.620{\tiny{$\pm$0.166}} & 0.620{\tiny{$\pm$0.082}} & 0.586{\tiny{$\pm$0.121}} & 0.614{\tiny{$\pm$0.196}} & 0.601{\tiny{$\pm$0.138}} & 0.573{\tiny{$\pm$0.163}} \\
Glass6 & 0.915{\tiny{$\pm$0.077}} & 0.884{\tiny{$\pm$0.085}} & 0.925{\tiny{$\pm$0.063}} & 0.939{\tiny{$\pm$0.057}} & \textbf{0.962{\tiny{$\pm$0.025}}} & 0.875{\tiny{$\pm$0.102}} & 0.939{\tiny{$\pm$0.046}} & 0.827{\tiny{$\pm$0.150}} \\
Haberman & 0.632{\tiny{$\pm$0.099}} & 0.548{\tiny{$\pm$0.116}} & 0.601{\tiny{$\pm$0.129}} & 0.662{\tiny{$\pm$0.091}} & 0.633{\tiny{$\pm$0.110}} & 0.586{\tiny{$\pm$0.131}} & \textbf{0.666{\tiny{$\pm$0.130}}} & 0.629{\tiny{$\pm$0.106}} \\
Iris0 & 0.993{\tiny{$\pm$0.011}} & 0.894{\tiny{$\pm$0.286}} & 0.997{\tiny{$\pm$0.009}} & 0.897{\tiny{$\pm$0.217}} & 0.950{\tiny{$\pm$0.158}} & 0.957{\tiny{$\pm$0.135}} & 0.998{\tiny{$\pm$0.004}} & \textbf{1.000{\tiny{$\pm$0.001}}} \\
New-Thyroid1 & 0.930{\tiny{$\pm$0.047}} & 0.802{\tiny{$\pm$0.223}} & 0.951{\tiny{$\pm$0.044}} & 0.962{\tiny{$\pm$0.035}} & 0.895{\tiny{$\pm$0.160}} & \textbf{0.973{\tiny{$\pm$0.065}}} & 0.888{\tiny{$\pm$0.300}} & 0.910{\tiny{$\pm$0.152}} \\
Pima & \textbf{0.758{\tiny{$\pm$0.015}}} & 0.722{\tiny{$\pm$0.062}} & 0.756{\tiny{$\pm$0.024}} & 0.753{\tiny{$\pm$0.032}} & 0.742{\tiny{$\pm$0.048}} & 0.757{\tiny{$\pm$0.026}} & 0.741{\tiny{$\pm$0.027}} & 0.745{\tiny{$\pm$0.027}} \\
Vehicle2 & \textbf{0.949{\tiny{$\pm$0.024}}} & 0.901{\tiny{$\pm$0.052}} & 0.948{\tiny{$\pm$0.021}} & 0.936{\tiny{$\pm$0.026}} & 0.928{\tiny{$\pm$0.029}} & 0.932{\tiny{$\pm$0.038}} & 0.864{\tiny{$\pm$0.049}} & 0.892{\tiny{$\pm$0.057}} \\
Vowel0 & 0.966{\tiny{$\pm$0.033}} & 0.929{\tiny{$\pm$0.054}} & \textbf{0.978{\tiny{$\pm$0.019}}} & 0.975{\tiny{$\pm$0.018}} & 0.966{\tiny{$\pm$0.015}} & 0.956{\tiny{$\pm$0.020}} & 0.847{\tiny{$\pm$0.157}} & 0.838{\tiny{$\pm$0.079}} \\
Wisconsin & 0.995{\tiny{$\pm$0.003}} & 0.912{\tiny{$\pm$0.214}} & \textbf{0.998{\tiny{$\pm$0.002}}} & 0.997{\tiny{$\pm$0.002}} & 0.995{\tiny{$\pm$0.006}} & 0.996{\tiny{$\pm$0.004}} & 0.993{\tiny{$\pm$0.014}} & 0.997{\tiny{$\pm$0.002}} \\
Yeast3 & \textbf{0.879{\tiny{$\pm$0.032}}} & 0.657{\tiny{$\pm$0.140}} & 0.861{\tiny{$\pm$0.070}} & 0.853{\tiny{$\pm$0.063}} & 0.846{\tiny{$\pm$0.091}} & 0.850{\tiny{$\pm$0.038}} & 0.793{\tiny{$\pm$0.104}} & 0.752{\tiny{$\pm$0.133}} \\
Glass0 & \textbf{0.801{\tiny{$\pm$0.042}}} & 0.767{\tiny{$\pm$0.042}} & 0.796{\tiny{$\pm$0.040}} & 0.793{\tiny{$\pm$0.051}} & 0.765{\tiny{$\pm$0.052}} & 0.745{\tiny{$\pm$0.089}} & 0.791{\tiny{$\pm$0.044}} \\
Glass1 & \textbf{0.725{\tiny{$\pm$0.036}}} & 0.718{\tiny{$\pm$0.054}} & 0.714{\tiny{$\pm$0.036}} & 0.700{\tiny{$\pm$0.039}} & 0.706{\tiny{$\pm$0.041}} & 0.635{\tiny{$\pm$0.052}} & 0.689{\tiny{$\pm$0.060}} \\
Glass2 & \textbf{0.622{\tiny{$\pm$0.110}}} & 0.620{\tiny{$\pm$0.166}} & 0.620{\tiny{$\pm$0.082}} & 0.586{\tiny{$\pm$0.121}} & 0.614{\tiny{$\pm$0.196}} & 0.601{\tiny{$\pm$0.138}} & 0.573{\tiny{$\pm$0.163}} \\
Glass6 & 0.915{\tiny{$\pm$0.077}} & 0.925{\tiny{$\pm$0.063}} & 0.939{\tiny{$\pm$0.057}} & \textbf{0.962{\tiny{$\pm$0.025}}} & 0.875{\tiny{$\pm$0.102}} & 0.939{\tiny{$\pm$0.046}} & 0.827{\tiny{$\pm$0.150}} \\
Haberman & 0.632{\tiny{$\pm$0.099}} & 0.601{\tiny{$\pm$0.129}} & 0.662{\tiny{$\pm$0.091}} & 0.633{\tiny{$\pm$0.110}} & 0.586{\tiny{$\pm$0.131}} & \textbf{0.666{\tiny{$\pm$0.130}}} & 0.629{\tiny{$\pm$0.106}} \\
Iris0 & 0.993{\tiny{$\pm$0.011}} & 0.997{\tiny{$\pm$0.009}} & 0.897{\tiny{$\pm$0.217}} & 0.950{\tiny{$\pm$0.158}} & 0.957{\tiny{$\pm$0.135}} & 0.998{\tiny{$\pm$0.004}} & \textbf{1.000{\tiny{$\pm$0.001}}} \\
New-Thyroid1 & 0.930{\tiny{$\pm$0.047}} & 0.951{\tiny{$\pm$0.044}} & 0.962{\tiny{$\pm$0.035}} & 0.895{\tiny{$\pm$0.160}} & \textbf{0.973{\tiny{$\pm$0.065}}} & 0.888{\tiny{$\pm$0.300}} & 0.910{\tiny{$\pm$0.152}} \\
Pima & \textbf{0.758{\tiny{$\pm$0.015}}} & 0.756{\tiny{$\pm$0.024}} & 0.753{\tiny{$\pm$0.032}} & 0.742{\tiny{$\pm$0.048}} & 0.757{\tiny{$\pm$0.026}} & 0.741{\tiny{$\pm$0.027}} & 0.745{\tiny{$\pm$0.027}} \\
Vehicle2 & \textbf{0.949{\tiny{$\pm$0.024}}} & 0.948{\tiny{$\pm$0.021}} & 0.936{\tiny{$\pm$0.026}} & 0.928{\tiny{$\pm$0.029}} & 0.932{\tiny{$\pm$0.038}} & 0.864{\tiny{$\pm$0.049}} & 0.892{\tiny{$\pm$0.057}} \\
Vowel0 & 0.966{\tiny{$\pm$0.033}} & \textbf{0.978{\tiny{$\pm$0.019}}} & 0.975{\tiny{$\pm$0.018}} & 0.966{\tiny{$\pm$0.015}} & 0.956{\tiny{$\pm$0.020}} & 0.847{\tiny{$\pm$0.157}} & 0.838{\tiny{$\pm$0.079}} \\
Wisconsin & 0.995{\tiny{$\pm$0.003}} & \textbf{0.998{\tiny{$\pm$0.002}}} & 0.997{\tiny{$\pm$0.002}} & 0.995{\tiny{$\pm$0.006}} & 0.996{\tiny{$\pm$0.004}} & 0.993{\tiny{$\pm$0.014}} & 0.997{\tiny{$\pm$0.002}} \\
Yeast3 & \textbf{0.879{\tiny{$\pm$0.032}}} & 0.861{\tiny{$\pm$0.070}} & 0.853{\tiny{$\pm$0.063}} & 0.846{\tiny{$\pm$0.091}} & 0.850{\tiny{$\pm$0.038}} & 0.793{\tiny{$\pm$0.104}} & 0.752{\tiny{$\pm$0.133}} \\
\bottomrule
\end{tabular}%
}
@@ -208,7 +148,7 @@ Yeast3 & \textbf{0.879{\tiny{$\pm$0.032}}} & 0.657{\tiny{$\pm$0.140}} & 0.
\subsection{CIFAR-10N: Label-Noise Robustness}\label{sec:results:cifar10n}
\Cref{tab:results:cifar10n} shows AUC under three noise levels on CIFAR-10N. Under \emph{clean} labels all methods reach AUC $\geq 0.972$, with \textsc{Roll}-AoC at the top (0.980). As noise increases the advantage of \textsc{Roll}-AoC grows: under \emph{aggre} noise it leads by 0.002 over the next-best method (GCE, 0.932) while other methods degrade more significantly, and under \emph{worse} noise, the margin widens to 0.012 over MAE (0.809) and 0.047 over BCE-W (0.738). Observing degredation w.r.t the clean scenario, we can see that \testxc{Roll}-AoC degraded the least (-0.159 relative to clean results) relative to other configurations, with the other least-degrading result being MAE (-0.167)
\Cref{tab:results:cifar10n} shows AUC under three noise levels on CIFAR-10N. Under \emph{clean} labels all methods reach AUC $\geq 0.972$, with \textsc{Roll}-AOC at the top (0.980). As noise increases the advantage of \textsc{Roll}-AOC grows: under \emph{aggre} noise it leads by 0.002 over the next-best method (GCE, 0.932) while other methods degrade more significantly, and under \emph{worse} noise, the margin widens to 0.012 over MAE (0.809) and 0.047 over BCE-W (0.738). Observing degradation relative to the clean scenario, \textsc{Roll}-AOC degraded the least ($-0.159$ relative to clean results), with MAE second ($-0.167$).
\begin{table}[htbp]
\centering
@@ -217,7 +157,7 @@ Yeast3 & \textbf{0.879{\tiny{$\pm$0.032}}} & 0.657{\tiny{$\pm$0.140}} & 0.
\resizebox{\textwidth}{!}{%
\begin{tabular}{lccccccc}
\toprule
Noise & \textsc{Roll}-AoC & BCE-W & Focal & Asym. & GCE & LibAUC & MAE \\
Noise & \textsc{Roll}-AOC & BCE-W & Focal & Asym. & GCE & LibAUC & MAE \\
\midrule
Clean & \textbf{0.980{\tiny{$\pm$0.001}}} & 0.977{\tiny{$\pm$0.002}} & 0.977{\tiny{$\pm$0.000}} & 0.972{\tiny{$\pm$0.003}} & 0.976{\tiny{$\pm$0.002}} & 0.979{\tiny{$\pm$0.000}} & 0.976{\tiny{$\pm$0.002}} \\
Aggre. & \textbf{0.934{\tiny{$\pm$0.002}}} & 0.924{\tiny{$\pm$0.003}} & 0.912{\tiny{$\pm$0.014}} & 0.910{\tiny{$\pm$0.004}} & 0.932{\tiny{$\pm$0.005}} & 0.921{\tiny{$\pm$0.004}} & 0.930{\tiny{$\pm$0.002}} \\
@@ -227,68 +167,31 @@ Worse & \textbf{0.821{\tiny{$\pm$0.001}}} & 0.738{\tiny{$\pm$0.009}} & 0.734{\ti
}
\end{table}
% TODO: Remove Roll-TPR column from tab:results:large and update the discussion accordingly.
% Roll-TPR is not the focus for these datasets and its results (e.g. 0.438 on Bank Marketing) distract from the main story.
\subsection{Large Real-World Datasets}\label{sec:results:large}
\subsection{Large Datasets}\label{sec:results:large}
% TODO: Reword Bank Marketing discussion after rerun — current results (Roll-AoC 0.511, Roll-TPR 0.438) are likely a bug. Update numbers and narrative once confirmed.
\Cref{tab:results:large} covers three large-scale datasets. On Credit Card Fraud \textsc{Roll}-TPR is the top performer (0.974) with \textsc{Roll}-AoC close behind (0.968), and both outperform all non-ROLL baselines. On Home Credit \textsc{Roll}-AoC edges LibAUC (0.712 vs.\ 0.711). On Bank Marketing, however, GCE achieves the highest AUC (0.746) while \textsc{Roll}-AoC and \textsc{Roll}-TPR underperform (0.511 and 0.438 respectively); the cause of this degradation on this dataset is an open question and a direction for future investigation.
\Cref{tab:results:large} covers the four large-scale datasets. On Bank Marketing, LibAUC leads (0.936) with \textsc{Roll}-AOC close behind (0.931), and both substantially outperform all other baselines. On Credit Card Fraud \textsc{Roll}-AOC is the top performer (0.968), outperforming all baselines. On Home Credit \textsc{Roll}-AOC edges LibAUC (0.712 vs.\ 0.711).
NIH ChestX-ray14 is qualitatively distinct from the other three: inputs are frozen DenseNet-121 embeddings from radiograph images rather than raw tabular features, making this a transfer-learning evaluation under naturally noisy labels~\cite{oakdenrayner2020exploring}.
\textsc{Roll}-AOC achieves the highest AUC ($0.839\pm0.001$), a clear margin above the next-best method GCE ($0.829\pm0.004$).
Standard cross-entropy variants trail substantially: BCE-W reaches $0.788\pm0.008$, Focal $0.798\pm0.009$, and Asym.\ $0.729\pm0.007$. MAE ($0.810\pm0.001$) and LibAUC ($0.835\pm0.002$) both fall short despite their respective noise-robust and AUC-optimising designs.
The ${\approx}5$~pp gap between \textsc{Roll}-AOC and BCE-W, consistent with the controlled KEEL noise experiment (\Cref{sec:results:keel}) and CIFAR-10N (\Cref{sec:results:cifar10n}), confirms that the noise-robustness advantage holds on real-world label noise. The result also shows that \textsc{Roll}-AOC generalises beyond from-scratch training: with embeddings frozen, ROLL operates entirely within the MLP head, fitting score distributions over a fixed representation.
\begin{table}[htbp]
\centering
\caption{Mean test AUC ($\pm$\,std, $n=5$ episodes) on large real-world datasets. Best per row in \textbf{bold}.}
\caption{Mean test AUC ($\pm$\,std, $n=5$ episodes) on large real-world datasets. Best per row in \textbf{bold}. ChestX-ray14 uses frozen DenseNet-121 embeddings rather than raw tabular features.}
\label{tab:results:large}
\resizebox{\textwidth}{!}{%
\begin{tabular}{lcccccccc}
\begin{tabular}{lccccccc}
\toprule
Dataset & \textsc{Roll}-AoC & \textsc{Roll}-TPR & BCE-W & Focal & Asym. & GCE & LibAUC & MAE \\
Dataset & \textsc{Roll}-AOC & BCE-W & Focal & Asym. & GCE & LibAUC & MAE \\
\midrule
Bank Marketing & 0.511{\tiny{$\pm$0.047}} & 0.438{\tiny{$\pm$0.044}} & 0.580{\tiny{$\pm$0.040}} & 0.573{\tiny{$\pm$0.044}} & 0.684{\tiny{$\pm$0.033}} & \textbf{0.746{\tiny{$\pm$0.022}}} & 0.652{\tiny{$\pm$0.031}} & 0.707{\tiny{$\pm$0.035}} \\
Credit Card Fraud & 0.968{\tiny{$\pm$0.001}} & \textbf{0.974{\tiny{$\pm$0.003}}} & 0.949{\tiny{$\pm$0.015}} & 0.942{\tiny{$\pm$0.020}} & 0.940{\tiny{$\pm$0.013}} & 0.948{\tiny{$\pm$0.014}} & 0.966{\tiny{$\pm$0.006}} & 0.954{\tiny{$\pm$0.009}} \\
Home Credit & \textbf{0.712{\tiny{$\pm$0.001}}} & 0.687{\tiny{$\pm$0.004}} & 0.693{\tiny{$\pm$0.002}} & 0.694{\tiny{$\pm$0.002}} & 0.688{\tiny{$\pm$0.003}} & 0.698{\tiny{$\pm$0.002}} & 0.711{\tiny{$\pm$0.001}} & 0.699{\tiny{$\pm$0.004}} \\
Bank Marketing & 0.931{\tiny{$\pm$0.002}} & 0.580{\tiny{$\pm$0.040}} & 0.573{\tiny{$\pm$0.044}} & 0.684{\tiny{$\pm$0.033}} & 0.746{\tiny{$\pm$0.022}} & \textbf{0.936{\tiny{$\pm$0.003}}} & 0.707{\tiny{$\pm$0.035}} \\
Credit Card Fraud & \textbf{0.968{\tiny{$\pm$0.001}}} & 0.949{\tiny{$\pm$0.015}} & 0.942{\tiny{$\pm$0.020}} & 0.940{\tiny{$\pm$0.013}} & 0.948{\tiny{$\pm$0.014}} & 0.966{\tiny{$\pm$0.006}} & 0.954{\tiny{$\pm$0.009}} \\
Home Credit & \textbf{0.712{\tiny{$\pm$0.001}}} & 0.693{\tiny{$\pm$0.002}} & 0.694{\tiny{$\pm$0.002}} & 0.688{\tiny{$\pm$0.003}} & 0.698{\tiny{$\pm$0.002}} & 0.711{\tiny{$\pm$0.001}} & 0.699{\tiny{$\pm$0.004}} \\
\midrule
ChestX-ray14 (Effusion) & \textbf{0.839{\tiny{$\pm$0.001}}} & 0.788{\tiny{$\pm$0.008}} & 0.798{\tiny{$\pm$0.009}} & 0.729{\tiny{$\pm$0.007}} & 0.829{\tiny{$\pm$0.004}} & 0.835{\tiny{$\pm$0.002}} & 0.810{\tiny{$\pm$0.001}} \\
\bottomrule
\end{tabular}%
}
\end{table}
\subsection{NIH ChestX-ray14}\label{sec:results:chestxray14}
\Cref{tab:results:chestxray14} reports test AUC on the ChestX-ray14 Effusion task.
\textsc{Roll}-AoC achieves the highest AUC ($0.839\pm0.001$), a clear margin above the
next-best methods GCE ($0.829\pm0.004$) and \textsc{Roll}-TPR ($0.828\pm0.002$).
Standard cross-entropy variants trail substantially: BCE-W reaches $0.788\pm0.008$,
Focal $0.798\pm0.009$, and Asym.\ $0.729\pm0.007$. MAE ($0.810\pm0.001$) benefits from
its noise-robust design but still falls behind the distributional losses. LibAUC results
are excluded due to an optimizer instability on this dataset (to be resolved in a
follow-up run).
The ${\approx}5$~pp gap between \textsc{Roll}-AoC and BCE-W, on a dataset whose labels
carry substantial organic NLP noise~\cite{oakdenrayner2020exploring}, is consistent with the findings from the controlled
KEEL noise experiment (\Cref{sec:results:keel}) and CIFAR-10N
(\Cref{sec:results:cifar10n}). The result also demonstrates that ROLL-AOC generalises
beyond from-scratch training: the DenseNet-121 embeddings are frozen, so ROLL operates
entirely within the MLP head, fitting score distributions over a fixed representation.
\begin{table}[htbp]
\centering
\caption{Mean test AUC ($\pm$\,std, $n=5$ episodes) on NIH ChestX-ray14 (Effusion
vs.\ rest, IR~${\approx}7.5$:1). Labels carry substantial organic NLP noise~\cite{oakdenrayner2020exploring}.
LibAUC excluded due to optimizer instability. Best in \textbf{bold}.}
\label{tab:results:chestxray14}
\begin{tabular}{lc}
\toprule
Method & AUC \\
\midrule
\textsc{Roll}-AoC & \textbf{0.839{\tiny{$\pm$0.001}}} \\
\textsc{Roll}-TPR & 0.828{\tiny{$\pm$0.002}} \\
GCE & 0.829{\tiny{$\pm$0.004}} \\
MAE & 0.810{\tiny{$\pm$0.001}} \\
Focal & 0.798{\tiny{$\pm$0.009}} \\
BCE-W & 0.788{\tiny{$\pm$0.008}} \\
Asym. & 0.729{\tiny{$\pm$0.007}} \\
LibAUC & 0.835{\tiny{$\pm$0.002}} \\
\bottomrule
\end{tabular}
\end{table}
content/introduction/introduction.tex
+5 -3
View File
@@ -4,14 +4,16 @@ Real-world data is hardly ever perfect. Many real-world datasets suffer a myriad
Gradient based learning methods, like those used to train neural networks, have trouble dealing with such datasets. Standard, per-sample losses (BCE, focal-loss), are confused easily by mislabeled points, as they produce a very large gradient which throws off training. Current attempts at noise-robust losses (GCE, MAE) help smooth these large gradients, yet still optimize on a per-sample basis, targeting per-sample accuracy rather than ranking quality --- leaving them misaligned with AUC even when they do reduce noise sensitivity.
In this field, optimizing directly for our desired metric (AUC) proves to be useful in dealing with noise. Per-sample losses help optimize each sample on their own, while AUC losses optimize the sample ranking amongst the whole. This helps them make smart tradeoffs while training, and accepting errors.
Direct AUC optimization is a natural direction here: ranking-based objectives are determined by the relative ordering of scores rather than their absolute values, making them structurally less sensitive to isolated mislabeled samples. Per-sample losses optimize each sample independently, while AUC losses optimize sample ranking across the whole population, allowing the model to make principled tradeoffs even in the presence of label errors.
\section{Contribution}
In this work we introduce the \textsc{ROLL} framework for ROC curve related losses and \textsc{ROLL}-AoC, a noise-robust loss that optimizes AUC \emph{directly}. We show that the loss is more robust to label noise than other methods, and how it handles class-imbalance naturally.
In this work we introduce the \textsc{ROLL} framework for ROC curve related losses and \textsc{ROLL}-AOC, a noise-robust loss that optimizes AUC \emph{directly}. We show that the loss is more robust to label noise than other methods, and how it handles class-imbalance naturally.
% TODO: Expand introduction to 4--6 pages. Currently too thin -- add motivation, concrete examples, and a clearer narrative arc before submission.
\section{Roadmap}
In this work, we will first review AUC optimization, KDE, imbalanced classification, label noise and existing methods for dealing with it in datasets. We then introduce \textsc{ROLL} and \textsc{ROLL}-AoC, formulate it using KDE distribution approximation, and offer derivations for use in gradient-based learning. Then, we show performance on real world data. Finally, we discuss advantages and disadvantages with regards to other methods, as well as offer avenues for future research.
In this work, we will first review AUC optimization, KDE, imbalanced classification, label noise and existing methods for dealing with it in datasets. We then introduce \textsc{ROLL} and \textsc{ROLL}-AOC, formulate it using KDE distribution approximation, and offer derivations for use in gradient-based learning. Then, we show performance on real world data. Finally, we discuss advantages and disadvantages with regards to other methods, as well as offer avenues for future research.
content/method/figures/roll_aoc_roc.pdf
BIN
View File
Binary file not shown.
content/method/figures/roll_aoc_roc.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 53 KiB

content/method/figures/roll_aoc_roc.py
+96
View File
@@ -0,0 +1,96 @@
"""
ROC curve illustration for ROLL-AOC.
Shows how ROLL-AOC converts the single-threshold ROLL into AUC optimization:
- ROC curve traced by sweeping threshold τ
- 10 evenly-spaced FPR sample points (as ROLL evaluates in miniature)
- Green shaded region = AUC (area under curve, what we maximize)
- Red shaded region = AOC = 1 − AUC (area over curve, the loss we minimize)
- Vertical segments = per-threshold loss 1 − TPR(αᵢ), one per sampled FPR
Run from the thesis root:
nix develop --command python3 content/method/figures/roll_aoc_roc.py
"""
import numpy as np
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from scipy.stats import norm
OUT_DIR = "content/method/figures"
# --- Parameters (same distributions as roll_principle.py) ------------------
mu0, sig0 = 0.0, 1.00
mu1, sig1 = 1.5, 1.00
# --- ROC curve -------------------------------------------------------------
# Sweep τ from high to low → FPR rises from 0 to 1
tau_grid = np.linspace(norm.ppf(0.9999, mu0, sig0),
norm.ppf(0.0001, mu0, sig0), 3000)
fpr_curve = 1.0 - norm.cdf(tau_grid, mu0, sig0)
tpr_curve = 1.0 - norm.cdf(tau_grid, mu1, sig1)
# Pin endpoints so the fill covers the full unit square correctly
fpr_full = np.concatenate([[0.0], fpr_curve, [1.0]])
tpr_full = np.concatenate([[0.0], tpr_curve, [1.0]])
auc = float(np.trapezoid(tpr_full, fpr_full))
# --- 10 sample FPR points: αᵢ = i/11 for i = 1…10 ------------------------
n_samples = 10
alpha_s = np.array([(i / (n_samples + 1)) for i in range(1, n_samples + 1)])
tpr_s = np.interp(alpha_s, fpr_full, tpr_full)
# --- Palette (matches roll_principle.py) -----------------------------------
C_POS = "#CC6633" # ROC curve (positive-class colour)
C_GREEN = "#44AA77" # AUC fill
C_RED = "#BB3333" # AOC fill / vertical segments
C_DOT = "#333333" # sample dots
# ── Figure ─────────────────────────────────────────────────────────────────
fig, ax = plt.subplots(figsize=(4.8, 4.4))
# Random-classifier diagonal
ax.plot([0, 1], [0, 1], color="#AAAAAA", linestyle="--",
linewidth=0.9, zorder=1, label="Random classifier")
# AUC — green fill under ROC curve
ax.fill_between(fpr_full, 0, tpr_full,
color=C_GREEN, alpha=0.25, zorder=2,
label=rf"AUC $\approx$ {auc:.2f} (maximize)")
# AOC — red fill above ROC curve
ax.fill_between(fpr_full, tpr_full, 1.0,
color=C_RED, alpha=0.22, zorder=2,
label=r"AOC $= 1-\mathrm{AUC}$ (minimize)")
# ROC curve itself
ax.plot(fpr_full, tpr_full, color=C_POS, linewidth=2.0, zorder=3,
label="ROC curve")
# Per-threshold loss segments: vertical from TPR(αᵢ) up to 1
for a_i, t_i in zip(alpha_s, tpr_s):
ax.plot([a_i, a_i], [t_i, 1.0],
color=C_RED, linewidth=1.0, alpha=0.55, zorder=4)
# Sample dots on the ROC curve
ax.scatter(alpha_s, tpr_s,
color=C_DOT, s=35, zorder=5,
label=r"Sampled $\alpha_i$ (ROLL evaluations)")
# --- Axes / labels ---------------------------------------------------------
ax.set_xlabel(r"FPR $(\alpha)$", fontsize=9)
ax.set_ylabel(r"TPR", fontsize=9)
ax.set_xlim(0.0, 1.0)
ax.set_ylim(0.0, 1.0)
ax.set_aspect("equal")
ax.tick_params(labelsize=8)
ax.spines[["top", "right"]].set_visible(False)
ax.legend(fontsize=7.5, loc="lower right", framealpha=0.88)
fig.tight_layout()
fig.savefig(f"{OUT_DIR}/roll_aoc_roc.pdf", bbox_inches="tight")
fig.savefig(f"{OUT_DIR}/roll_aoc_roc.png", bbox_inches="tight", dpi=180)
print(f"Saved to {OUT_DIR}/roll_aoc_roc.{{pdf,png}}")
print(f"AUC = {auc:.4f} AOC = {1-auc:.4f}")
content/method/method.tex
+31 -52
View File
@@ -73,7 +73,7 @@ In practice, however, making these evaluations differentiable is not straightfor
% common surrogates (AUC proxies, hinge-based losses) only indirectly target
% the operating point a practitioner cares about.
Directly optimizing TPR for FPR, or vice versa, is an indefferentiable problem.
Directly optimizing TPR for FPR, or vice versa, is a non-differentiable problem.
For this sub-section, we will reffer to an example of maximizing TPR given a fixed FPR,
though this can be reversed.
@@ -87,7 +87,7 @@ There are two inherent steps built-in to our methods:
\begin{itemize}
\item Based on fixed rate, calculate threshold that gives said rate.
For example, when fixing FPR = $\alpha$, find some threshold $\tau$ s.t. $\mathbb{E}[1_{f_{\theta}\mathbf{x}_i > \tau | y_i = 0}] = \alpha$
For example, when fixing FPR = $\alpha$, find some threshold $\tau$ such that $\mathbb{E}[\mathbf{1}[f_\theta(\mathbf{x}_i) > \tau \mid y_i = 0]] = \alpha$
\item Given the calculated threshold, calculate the rate we would like to maximize
(in our example, maximize TPR)
\end{itemize}
@@ -113,8 +113,8 @@ This necessitates a different approach, to make these calculations differentiabl
The way to overcome this problem in differentiability is not to calculate the TPR and FPR
directly, but to estimate them by fitting probability distributions to the score output of the model at
every step. Contiguous probability distributions are contiguous w.r.t their defining parameters, and a derivation of the parameters w.r.t the distributions themselves can be made.
Additionally, we can calculate the TPR, FPR given the probability distributions CDF and ICDF functions. If we can gaurantee these functions to be differentiable, which proved to be the main challenge of this work, we can gaurantee the entire process to be differentiable. Once we do so, a derivative of the scores themselves w.r.t. the loss can be acheived.
every step. Continuous probability distributions are continuous with respect to their defining parameters, and a derivation of the parameters with respect to the distributions themselves can be made.
Additionally, we can calculate the TPR, FPR given the probability distributions CDF and ICDF functions. If we can guarantee these functions to be differentiable, which proved to be the main challenge of this work, we can guarantee the entire process to be differentiable. Once we do so, a derivative of the scores themselves with respect to the loss can be achieved.
\subsection{Per-Threshold ROLL Loss}
@@ -163,7 +163,7 @@ Using this notation, at a threshold $\tau$:
\begin{align}
\operatorname{TPR}(\tau) &= 1 - \hat{F}_1(\tau), \label{eq:tpr-cdf} \\
\operatorname{FPR}(\tau) &= 1 - \hat{F}_0(\tau). \label{eq:fpr-cdf}
\end{align}t
\end{align}
Let $\hat{F}_1^{-1}$ and $\hat{F}_0^{-1}$ denote the quantile functions of $P_{\hat{\phi}_1}$ and
$P_{\hat{\phi}_0}$, respectively. The threshold achieving a target TPR of $\alpha$ follows directly
@@ -273,8 +273,7 @@ $1 - \operatorname{AUC}$. Since $1 - \operatorname{TPR}(\alpha) =
\begin{equation}
\mathcal{L}_{\text{AOC}} = \int_0^1\mathcal{L}_{\text{ROLL-TPR@FPR}}(\alpha)\,d\alpha.
\end{equation}
Direct integration is not tractable, but the integral is well-approximated by a Riemann
sum. Evaluating at 99 evenly-spaced FPR values $\alpha = i/100$ for $i = 1, \ldots, 99$
Direct integration is not tractable, but the integral is well-approximated by averaging out sampled points. Evaluating at 99 evenly-spaced FPR values $\alpha = i/100$ for $i = 1, \ldots, 99$
(excluding the degenerate endpoints 0 and 1), the ROLL-AOC loss is:
\begin{equation}
\mathcal{L}_{\text{ROLL-AOC}} = \frac{1}{99}\sum_{i=1}^{99}\mathcal{L}_{\text{ROLL-TPR@FPR}}\!\left(\frac{i}{100}\right)
@@ -291,6 +290,25 @@ The concrete per-threshold gradient for the KDE instantiation is derived in
\Cref{sec:roll-instantiations}; gradients for the parametric cases are in
\Cref{appendix:gauss-grad,appendix:beta-roll}.
\Cref{fig:roll-aoc-roc} gives a geometric view of this objective.
\begin{figure}[H]
\centering
\includegraphics[width=0.62\textwidth]{content/method/figures/roll_aoc_roc.pdf}
\caption{Geometric interpretation of ROLL-AOC. The ROC curve (orange) maps
each FPR value $\alpha$ to $\operatorname{TPR}(\alpha)$.
\textbf{Green}: area under the curve (AUC), the quantity we maximize.
\textbf{Red}: area over the curve
($\mathrm{AOC} = 1 - \mathrm{AUC} = \mathcal{L}_{\text{ROLL-AOC}}$),
the loss we minimize.
Dots mark 10 evenly-spaced FPR sample points $\alpha_i$; vertical
segments show the per-threshold loss contribution
$1 - \operatorname{TPR}(\alpha_i) = \mathcal{L}_{\text{ROLL-TPR@FPR}}(\alpha_i)$
at each point. Averaging these contributions (99 in practice, 10 shown
for clarity) gives the value of \eqref{eq:roll-aoc}.}
\label{fig:roll-aoc-roc}
\end{figure}
%------------------------------------------------
@@ -336,7 +354,7 @@ As before, to calculate the loss, we calculate:
\mathcal{L}_{\text{ROLL-TPR@FPR}}^{\text{KDE}}(f_{\theta}(\mathcal{B}) ; \alpha) = \hat{F}_1(\hat{F}_0^{-1}(1-\alpha))
\]
To calculate this, we first define the kernel function as $\sigma '$, and the CDF of the kernel function as $\sigma$. The approach used here is to define $\sigma$ as a sigmoid function,.
To calculate this, we first define the kernel function as $\sigma '$, and the CDF of the kernel function as $\sigma$. The approach used here is to define $\sigma$ as a sigmoid function.
\begin{equation}
\sigma(x ; v) = \left( 1 + \exp(-vx)\right)^{-1} \qquad \sigma'(x ; v) = \frac{v \cdot \exp(-v \cdot x)}{(1 + \exp(-vx))^2}
@@ -358,9 +376,9 @@ We use bisection, which requires no derivative evaluation and is guaranteed to c
for any strictly monotone CDF; the bracket construction and iteration are detailed in
Appendix~\ref{appendix:kde-bisection}.
We can now relatively easily calculate $\hat{F}_1(\tau ; \mathbf{X}_1)$ using the formulat for $\hat{F}_{KDE}(\tau ; \mathbf{X})$ above.
We can now trivially compute $\hat{F}_1(\tau ; \mathbf{X}_1)$ using the formula for $\hat{F}_{KDE}(\tau ; \mathbf{X})$ above.
Importantly, we must save the calculation of $\tau$ for the backward derivation, as well as the calculated bandwidths $v_1, v_0$.
Note that $\tau$ is cached for use in the backward pass, along with the computed bandwidths $v_1, v_0$.
\subsubsection{Gradient Derivation}
@@ -408,7 +426,7 @@ A structural consequence of the ROLL formulation is that gradient mass is automa
balanced between the two classes, regardless of their relative sizes in the batch.
This stands in contrast to standard losses such as cross-entropy, where the total gradient
contribution from each class is proportional to its count, making heavy class imbalance
a direct source of biased updates. Therefor, instead of needing to compensate for this
a direct source of biased updates. Therefore, instead of needing to compensate for this
with loss-weighting per-class, fitting updates are made based equally off the results
of both classes.
@@ -430,44 +448,7 @@ independently of $|\mathcal{B}_0|$ and $|\mathcal{B}_1|$.
\end{proposition}
\begin{proof}
We treat each class separately via a uniform-shift argument.
\medskip\noindent\textit{Sum over $\mathcal{B}_0$.}
From \Cref{eq:roll-gradient}, for $y_i = 0$:
\[
\frac{\partial \mathcal{L}}{\partial f_\theta(\mathbf{x}_i)}
= \frac{\partial \hat{F}_1(\tau)}{\partial \tau}
\cdot \frac{\partial \tau}{\partial f_\theta(\mathbf{x}_i)}
\]
We claim $\sum_{\mathbf{x}_i \in \mathcal{B}_0}\frac{\partial \tau}{\partial f_\theta(\mathbf{x}_i)} = 1$.
Shift all negative scores uniformly: $f_\theta(\mathbf{x}_i) \to f_\theta(\mathbf{x}_i) + \delta$ for
$\mathbf{x}_i \in \mathcal{B}_0$. By translation invariance, $\hat{F}_0$ simply translates, so
$\tau = \hat{F}_0^{-1}(1-\alpha)$ shifts by exactly $\delta$. Differentiating with respect to $\delta$ at
$\delta = 0$:
\[
1 = \frac{\mathrm{d}\tau}{\mathrm{d}\delta}\bigg|_{\delta=0}
= \sum_{\mathbf{x}_i \in \mathcal{B}_0} \frac{\partial \tau}{\partial f_\theta(\mathbf{x}_i)}
\]
Summing the $y_i = 0$ gradients therefore gives:
\begin{equation}
\sum_{\mathbf{x}_i \in \mathcal{B}_0}
\frac{\partial \mathcal{L}}{\partial f_\theta(\mathbf{x}_i)}
= \frac{\partial \hat{F}_1(\tau)}{\partial \tau} \cdot 1
= \frac{\partial \hat{F}_1(\tau)}{\partial \tau}
\label{eq:grad-balance-b0}
\end{equation}
\medskip\noindent\textit{Sum over $\mathcal{B}_1$.}
For $y_i = 1$, $\frac{\partial \mathcal{L}}{\partial f_\theta(\mathbf{x}_i)} = \frac{\partial \hat{F}_1(\tau)}{\partial f_\theta(\mathbf{x}_i)}$.
Shifting all positive scores by $\delta$ and applying translation invariance of $\hat{F}_1$:
$\hat{F}_1(\tau\,;\,\mathbf{s}+\delta\mathbf{1}) = \hat{F}_1(\tau-\delta\,;\,\mathbf{s})$.
Differentiating at $\delta = 0$:
\[
\sum_{\mathbf{x}_i \in \mathcal{B}_1}
\frac{\partial \hat{F}_1(\tau)}{\partial f_\theta(\mathbf{x}_i)}
= -\frac{\partial \hat{F}_1(\tau)}{\partial \tau}
\]
Combining with \eqref{eq:grad-balance-b0} gives \eqref{eq:grad-balance}.
See \Cref{appendix:grad-balance}.
\end{proof}
The translation-invariance assumption holds for both the Gaussian distribution,
@@ -479,7 +460,7 @@ is equivalent to an equal shift in $\tau$. The condition is necessary: without i
score shift need not produce a rigid translation of $\hat{F}_k$, so the shift argument in the
proof breaks down and the gradient sums need not be equal.
The Beta distribution only accepts values in the interval $\[0, 1\]$. Since it
The Beta distribution only accepts values in the interval $[0, 1]$. Since it
is mapped only over an interval, the translation-invariance assumption does not hold
for this distribution.
@@ -625,5 +606,3 @@ distribution, producing very low gradient magnitudes and stalling convergence.
The solution was to use a bandwidth scheduler. Starting with large bandwidths resulted in gradients that spread out more evenly across all scores, and fewer non-convergence runs occurred. The bandwidth scheduler artificially scales up the bandwidth at the beginning to speed up initial training, and then tapers back to restore the intended bandwidth in order to maximise results.
% TODO: algorithm box — full training loop (forward → ROLL loss → custom backward → weight update)
content/related_work/related_work.tex
+12 -8
View File
@@ -62,9 +62,9 @@ where $x_1 \sim \mathcal{D}_1$ and $x_0 \sim \mathcal{D}_0$ are drawn independen
\subsection{AUC as a metric of model performance}
The AUC is a direct representation of the performance of a classifier, as opposed to a loss. There are instances where the loss may be minimized, yet seperation might be minimal. These two metrics might correlate, but do not necessarily correspond.
The AUC is a direct representation of the performance of a classifier, as opposed to a loss. There are instances where the loss may be minimized, yet separation might be minimal. These two metrics might correlate, but do not necessarily correspond.
An example of this is a result for which all true samples receive a score of $0.51$ and all false samples receie a score of $0.49$. Using many losses, including BCE, this would result in a relatively high loss value. On this score however, the AUC is $1$. The model perfectly seperates the samples, so further training is not necessary.
An example of this is a result for which all true samples receive a score of $0.51$ and all false samples receive a score of $0.49$. Using many losses, including BCE, this would result in a relatively high loss value. On this score however, the AUC is $1$. The model perfectly separates the samples, so further training is not necessary.
Direct AUC optimization can be ideal with optimizing for imbalanced datasets. With most losses such as BCE or focal, the loss is calculated per-sample and then summed (or averaged). In order to properly train for both true and false populations, the scores are weighted, so that equal weight falls on both the true and the false populations. With direct AUC optimization, the imbalance does not affect the calculation, and the reweighting can be forgone.
@@ -72,9 +72,9 @@ The noise-robustness properties of ranking-based objectives are discussed in \Cr
\subsection{AUC optimization in literature}
While AUC as a metric for model performance has existed for a while, methods to optimize for it directly are relatively recent. At it's core, it is a non-trivial problem to solve. Existing efforts to directly optimize AUC have been based on the Wilcoxon-Mann-Whitney interpretation of AUC. However, directly calculating the afformentioned probability of a random positive score behing higher than a negative one introduces two complications. The first is that one must iterate over $n^2$ pairs to cover every posible pair. Additionally, the pairwise function one must evaluate ( $\mathbb{P}(f(x_1) > f(x_0))$ ) is a non-contiguous stepwise function, and is therefor on-differentiable.
While AUC as a metric for model performance has existed for a while, methods to optimize for it directly are relatively recent. At it's core, it is a non-trivial problem to solve. Existing efforts to directly optimize AUC have been based on the Wilcoxon-Mann-Whitney interpretation of AUC. However, directly calculating the aforementioned probability of a random positive score being higher than a negative one introduces two complications. The first is that one must iterate over $n^2$ pairs to cover every possible pair. Additionally, the pairwise function one must evaluate ($\mathbb{P}(f(x_1) > f(x_0))$) is a discontinuous stepwise function, and is therefore non-differentiable.
The non-contiguous stepwise nature of the evaluated pairwise function is generally fixed by using a surrogate, either sigmoid-like or hinge functions are applied to the score difference~\cite{cortes2003auc}.
The discontinuous stepwise nature of the evaluated pairwise function is generally addressed by using a surrogate, either sigmoid-like or hinge functions are applied to the score difference~\cite{cortes2003auc}.
\citet{yuan2021auc} proposed a compositional stochastic gradient framework that reformulates AUC maximization as a min-max saddle-point problem, enabling mini-batch training without maintaining all pairwise scores in memory. Their PESG optimizer and the LibAUC library are the primary practical tools for deep AUC maximization and serve as the main AUC baseline in this thesis.
% ============================================================
@@ -182,7 +182,11 @@ Intermediate values therefore trade noise tolerance for faster convergence.
We evaluate GCE at $q = 0.7$ as a baseline in the Experiments chapter.
\citet{zhou2021asymmetric} extend the theoretical framework to asymmetric noise, introducing
the \emph{asymmetry ratio} to quantify how noise tolerance degrades when flip rates differ
between classes.
between classes, and propose an asymmetric loss that applies distinct focusing parameters to
positive and negative samples: easy negatives are downweighted more aggressively than positives,
counteracting both gradient dominance from numerous easy negatives in imbalanced settings and
the increased noise sensitivity that comes with symmetric focusing.
This asymmetric loss is evaluated as a baseline (\textbf{Asym.}) in \Cref{chapter:experiments}.
\citet{feng2020can} confirm that standard cross-entropy cannot be made noise-tolerant
through simple modification, establishing robust losses as a genuinely distinct family
rather than a patch on existing objectives.
@@ -265,13 +269,13 @@ This gives the KDE cumulative distribution function a closed form as well:
It is rare for a dataset to have an equal amount of true and false samples. Class imbalance
is something that must be taken into account during training on most datasets.
Extreme imbalance exacerbates this issue further - for a dataset with an imbalance ratio of $100$,
a \emph{constant} classifier (outputting only the most probable class) acheives a very good result.
a \emph{constant} classifier (outputting only the most probable class) achieves a very good result.
Without correction, many learning methods will find a local minima in a relatively trivial solution, such
as constant output, and have a hard time adjusting. Even if not in this local minima, less emphasis is put
on the rare class during training, as it contributes less to the loss metric, and the model tends to
generalize on the rare class poorly.
Dealing with this depends on the loss used. For most losses, for which the loss for each sample is calculated seperately,
Dealing with this depends on the loss used. For most losses, for which the loss for each sample is calculated separately,
the solution is to simply weight the losses for each class differently,
by multiplying the loss for the rare class
so that it contributes equally. Other approaches using pairwise losses, such as those implemented in LibAUC \cite{yuan2021auc}, naturally weight positive and negative contributions to learning through their implementation.
@@ -323,7 +327,7 @@ The NP lemma provides the theoretically optimal solution but assumes the class-c
$p(\mathbf{x} \mid y=0)$ and $p(\mathbf{x} \mid y=1)$ are known.
In practice these densities must be estimated from data.
A model must be fit to the data in order to estimate these probabilities, meaning that
there is no gaurantee on optimality.
there is no guarantee on optimality.
Further work regarding \emph{Neyman-Pearson classification} adapts the NP
principle to the finite-sample setting using nonparametric methods
main.tex
+18 -13
View File
@@ -192,7 +192,7 @@
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%Title and names for the article
\newcommand{\thesistitle}{\textbf{Noise-Robust Gradient-Based Learning Utilizing Rate Optimized Likelyhood Loss}}
\newcommand{\thesistitle}{\textbf{Noise-Robust Gradient-Based Learning Utilizing Rate Optimized Likelihood Loss}}
\newcommand{\thesistitlehe}{\textbf{למידה עמידה לרעש מבוססת גרדיאנט תוך שימוש בפונקציית הפסד קירוב-הסתברות}}
\newcommand{\thesisauthorname}{\textbf{Aner Zakobar}}
@@ -256,46 +256,51 @@
\begin{appendices}
\chapter{Some title for apendix A}
\label{appendix:numerically_distinguished_comp}
\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח א}{\quad\qquadדרישת תאימות כלשהי}}{{\beginL\thepage\endL}}}
\input{content/appendixs/appendix_a}
\chapter{Gaussian ROLL: Forward Pass and Gradient Derivation}
\label{appendix:gauss-grad}
\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ב}{\quad\qquadנגזרת גאוסיאנית של ROLL}}{{\beginL\thepage\endL}}}
\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח א}{\quad\qquadנגזרת גאוסיאנית של ROLL}}{{\beginL\thepage\endL}}}
\input{content/appendixs/appendix_gauss_grad}
\chapter{Beta ROLL: Forward Pass and Gradient}
\label{appendix:beta-roll}
\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ג}{\quad\qquadBeta ROLL: אמידה ונגזרת}}{{\beginL\thepage\endL}}}
\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ב}{\quad\qquadBeta ROLL: אמידה ונגזרת}}{{\beginL\thepage\endL}}}
\input{content/appendixs/appendix_beta_roll}
\chapter{KDE ROLL: Gradient Derivation}
\label{appendix:kde-grad}
\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ד}{\quad\qquadנגזרת KDE של ROLL}}{{\beginL\thepage\endL}}}
\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ג}{\quad\qquadנגזרת KDE של ROLL}}{{\beginL\thepage\endL}}}
\input{content/appendixs/appendix_kde_grad}
\chapter{KDE ROLL: Sigmoid Kernel — Explicit Equations}
\label{appendix:kde-sigmoid}
\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ה}{\quad\qquadKDE ROLL: ניסוח מפורש עם גרעין סיגמואיד}}{{\beginL\thepage\endL}}}
\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ד}{\quad\qquadKDE ROLL: ניסוח מפורש עם גרעין סיגמואיד}}{{\beginL\thepage\endL}}}
\input{content/appendixs/appendix_kde_sigmoid}
\chapter{KDE ROLL: Bisection for CDF Inversion}
\label{appendix:kde-bisection}
\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ו}{\quad\qquadKDE ROLL: היפוך CDF בשיטת חצייה}}{{\beginL\thepage\endL}}}
\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ה}{\quad\qquadKDE ROLL: היפוך CDF בשיטת חצייה}}{{\beginL\thepage\endL}}}
\input{content/appendixs/appendix_kde_bisection}
\chapter{KDE ROLL: Newton--Raphson Inversion (Superseded)}
\label{appendix:kde-nr}
\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ז}{\quad\qquadKDE ROLL: היפוך ניוטון-רפסון}}{{\beginL\thepage\endL}}}
\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ו}{\quad\qquadKDE ROLL: היפוך ניוטון-רפסון}}{{\beginL\thepage\endL}}}
\input{content/appendixs/appendix_kde_nr}
\chapter{ROLL Loss: Implementation Listing}
\label{appendix:roll-code}
\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ח}{\quad\qquadROLL: קוד מימוש}}{{\beginL\thepage\endL}}}
\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ז}{\quad\qquadROLL: קוד מימוש}}{{\beginL\thepage\endL}}}
\input{content/appendixs/appendix_roll_code}
\chapter{KEEL Benchmark: Noise Tables at 5\% and 10\%}
\label{appendix:keel-noise}
\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ח}{\quad\qquadKEEL: טבלאות רעש 5\% ו-10\%}}{{\beginL\thepage\endL}}}
\input{content/appendixs/appendix_keel_noise}
\chapter{Proof of the Gradient Balance Proposition}
\label{appendix:grad-balance}
\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ט}{\quad\qquadהוכחת טענת איזון הגרדיאנט}}{{\beginL\thepage\endL}}}
\input{content/appendixs/appendix_grad_balance}
\end{appendices}
%~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~%