20 lines
2.9 KiB
TeX
20 lines
2.9 KiB
TeX
\section{Problem Statement}
|
|
|
|
Real-world data is hardly ever perfect. Many real-world datasets suffer a myriad of problems. One such common problem is improper labeling. While we would like our models to learn based on an exact ground truth, often the labels with which we train our classifiers have errors themselves. Label-noise plagues many datasets for a variety of reasons. In datasets such as CIFAR-10N, labels were collected from crowdworker annotators, who are often prone to mistakes. In ChestXray14, a dataset of medical chest X-rays, labels were given by a natural language processing algorithm, since labelling each image by hand was deemed impractical. Such automatic labeling gave way to relatively high label error rates. In other cases, labels may only be assigned to part of the data, due to high effort needed to label, or the labels might be based on other data entirely, causing some misalignment.
|
|
|
|
Gradient based learning methods, like those used to train neural networks, have trouble dealing with such datasets. Mislabeled data points are often assigned a score that is very far from the label itself. While this makes sense, the data point is very different from the class it is assigned, this mismatch can create a very large corrective gradient for training, a corrective gradient steering the model in the wrong direction. Current attempts at noise-robust losses (GCE, MAE) help smooth these large gradients, yet still optimize on a per-sample basis, targeting per-sample accuracy rather than ranking quality. This leaves them misaligned with AUC even when they do reduce noise sensitivity.
|
|
|
|
Direct AUC optimization is a natural direction here: ranking-based objectives are determined by the relative ordering of scores rather than their absolute values, making them less sensitive to isolated mislabeled samples. Per-sample losses optimize each sample independently, while AUC losses optimize sample ranking across the whole population, allowing the model to make principled tradeoffs even in the presence of label errors.
|
|
|
|
\section{Contribution}
|
|
|
|
In this work we introduce the \textsc{ROLL} framework for ROC curve related losses and \textsc{ROLL}-AOC, a noise-robust loss that optimizes AUC \emph{directly}. We show that the loss is more robust to label noise than other methods, and how it handles class-imbalance naturally.
|
|
|
|
% TODO: Expand introduction to 4--6 pages. Currently too thin -- add motivation, concrete examples, and a clearer narrative arc before submission.
|
|
|
|
\section{Roadmap}
|
|
|
|
In this work, we will first review AUC optimization, KDE, imbalanced classification, label noise and existing methods for dealing with it in datasets. We then introduce \textsc{ROLL} and \textsc{ROLL}-AOC, formulate it using KDE distribution approximation, and offer derivations for use in gradient-based learning. Then, we show performance on real world data. Finally, we discuss advantages and disadvantages with regards to other methods, as well as offer avenues for future research.
|
|
|
|
|