Wrote stuff?

Main_pages/abstract.tex

@@ -25,7 +25,13 @@
     \textbf{Abstract}
 \end{center}
 
-In this paper we present ...
+
+Real datasets used to train models are often plagued by
+label noise, where a fraction of training labels are incorrect. Standard per-sample losses such as binary cross-entropy are sensitive to mislabeled examples and optimize per-sample accuracy rather than ranking quality, making them misaligned with the Area Under the ROC Curve (AUC), the standard metric for imbalanced classification. Noise-robust losses (GCE, MAE) address the sensitivity problem but remain per-sample and inherit the same metric mismatch. Existing AUC-maximizing methods improve on this by directly targeting ranking quality, but rely on pairwise surrogate losses that optimize for AUC indirectly.
+
+In this work we introduce \textsc{ROLL} (Rate Optimized Likelihood Loss), a framework for differentiable ROC-curve optimization via output score distribution fitting. The primary instantiation, \textsc{ROLL}-AoC, approximates AUC as the mean TPR sampled across a range of FPR thresholds. This loss utilizes a Kernel Density Estimation to approximate the distribution of output scores at each epoch, and optimizes it directly via gradient descent with a custom backward pass. Although noise robustness is not an explicit design objective, \textsc{ROLL}-AoC exhibits strong implicit resistance to label noise.
+
+We evaluate on twelve KEEL benchmark datasets under label noise, CIFAR-10N under real crowdworker annotation noise, three large-scale tabular datasets, and NIH ChestX-ray14 with organic NLP-extracted label noise. Across all benchmarks, \textsc{ROLL}-AoC is competitive under clean labels and leads under high noise rates, outperforming dedicated noise-robust losses.
 
 
 

Main_pages/abstract_heb.tex

@@ -29,6 +29,10 @@
     \textbf{תקציר}
 \end{center}
 
-בלה בלה בלה בלה בלה בלה .
+מערכי נתונים אמיתיים המשמשים לאימון מודלים סובלים לעיתים קרובות מרעש תוויות, שבו חלק מתוויות האימון שגויות. פונקציות הפסד סטנדרטיות המחושבות לכל דגימה בנפרד, כגון binary cross-entropy, רגישות לדגימות המתויגות באופן שגוי. הן נוטות לטייב את המודל כאשר כל דגימה נלקחת בנפרדת, במקום לטייב את דיגורן ככלל, מה שמנתק אותן מהמדד הרצוי, שטח מתחת לעקומת ROC‏ (AUC), המדד הסטנדרטי לסיווג. פונקציות הפסד עמידות לרעש (GCE ,MAE) מטפלות ברגישות זו, אך נותרות מחושבות לדגימה בודדת ויורשות את אותו פער במדד. שיטות קיימות לאופטימיזציה של AUC משפרות זאת על ידי כיוון לאיכות דירוג, אך מסתמכות על פונקציות הפסד-תחליף זוגיות ומטייבות את AUC באופן עקיף.
 
-\selectlanguage{english}
+בעבודה זו אנו מציגים את \textenglish{\textsc{ROLL}} ‏(Rate Optimized Likelihood Loss), מסגרת לאופטימיזציה דיפרנציאבילית של עקומת ROC באמצעות התאמת התפלגות ציוני הפלט של המודל. המימוש העיקרי, \textenglish{\textsc{ROLL}-AoC}, מקרב את AUC כממוצע TPR הנדגם על פני טווח ערכי FPR. פונקציית הפסד זו משתמשת \textenglish{Kernel Density Estimation (KDE)} לקירוב התפלגות ציוני הפלט בכל שלב אימון, ומטייבת עבורה באופן ישיר. למרות שעמידות לרעש אינה מטרת עיצוב מפורשת, \textenglish{\textsc{ROLL}-AoC} מפגין עמידות משמעותית ומובהקת לרעש תוויות.
+
+אנו מעריכים את השיטה על שנים-עשר מערכי נתונים מהאוסף KEEL תחת רעש תוויות, CIFAR-10N תחת רעש אנוטציה אנושי אמיתי, שלושה מערכי נתונים טבלאיים בקנה מידה גדול, ו-NIH ChestX-ray14 עם רעש תוויות אורגני שנוצר על ידי אלגוריתם עיבוד שפה טבעית. על פני כלל הבנצ'מרקים, \textenglish{\textsc{ROLL}-AoC} תחרותי בנתונים נקיים ומוביל בשיעורי רעש גבוהים, ועולה על פונקציות הפסד ייעודיות עמידות לרעש.
+
+\selectlanguage{english}

Main_pages/acknowledgements.tex

@@ -1,3 +1,8 @@
 \chapter*{Acknowledgements}
 
-First and foremost, I would like to thank my inspiration Alan Turing...
+With all due respect to Alan Turing, I would first like to thank my wife, Chen, for her everlasting support and love.
+Thank you to my daughters, Ella and Tom, for being the lights of my life, and, despite their best efforts, not breaking my computer.
+I would like to thank my parents, Valerie and Yair, for teaching me the value of education, hard work, and for giving me lunch money.
+Thank you to my sister Niva for always having my back.
+Thank you to my mentors, Prof. Rokach and Prof. Kontorovich, for your patience and guidance.
+

content/appendixs/appendix_beta_roll.tex

@@ -0,0 +1,59 @@
+We instantiate the general ROLL framework (\Cref{sec:roll-formulation}) with Beta-distributed
+scores.  Beta ROLL applies when model scores are constrained to the open interval $(0,1)$,
+as is the case when $f_\theta$ uses a sigmoid output activation.  The Beta distribution is
+supported on $(0,1)$ and its two shape parameters allow it to capture a wide range of
+unimodal and skewed score distributions.
+
+\paragraph{Forward pass.}
+
+Given score vectors $\mathbf{s}^{(k)} \in (0,1)^{|\mathcal{B}_k|}$ for class $k$, let
+$\mathbf{t}^{(k)} = \mathbf{1} - \mathbf{s}^{(k)}$ denote the complementary scores.
+For any vector $\mathbf{v}$, write $\bar{v} = \frac{1}{|\mathbf{v}|}\sum_i v_i$.
+We estimate the shape parameters using the closed-form log-moment estimator
+of~\cite{chen2022beta}: define for each class $k$,
+
+\begin{equation}
+  D_k = \overline{s^{(k)}\!\ln s^{(k)}} - \bar{s}^{(k)}\cdot\overline{\ln s^{(k)}}
+      + \overline{t^{(k)}\!\ln t^{(k)}} - \bar{t}^{(k)}\cdot\overline{\ln t^{(k)}}
+  \label{eq:beta-denom}
+\end{equation}
+
+\begin{equation}
+  \hat{a}_k = \frac{\bar{s}^{(k)}}{D_k}, \qquad \hat{b}_k = \frac{\bar{t}^{(k)}}{D_k}
+  \label{eq:beta-params}
+\end{equation}
+
+These estimators follow from two simultaneous moment conditions:
+$\mathbb{E}[X] = a/(a+b)$ and
+$\operatorname{Cov}(X,\ln X) + \operatorname{Cov}(Y,\ln Y) = 1/(a+b)$,
+where $Y=1-X\sim\operatorname{Beta}(b,a)$.
+Replacing expectations by sample means and solving in closed form yields~\eqref{eq:beta-params}.
+
+The CDF of $\operatorname{Beta}(\hat{a}_k,\hat{b}_k)$ at $x\in(0,1)$ is the
+regularised incomplete beta function:
+\begin{equation}
+  \hat{F}_k(x) = I_x(\hat{a}_k,\hat{b}_k)
+               = \frac{B(x;\,\hat{a}_k,\hat{b}_k)}{B(\hat{a}_k,\hat{b}_k)},
+  \qquad B(x;\,a,b) = \int_0^x u^{a-1}(1-u)^{b-1}\,\mathrm{d}u
+  \label{eq:beta-cdf}
+\end{equation}
+
+Substituting into the general ROLL objective \eqref{eq:roll-tpr-at-fpr}:
+\begin{equation}
+  \mathcal{L}_{\text{ROLL-TPR@FPR}}^{\text{BETA}}(f_\theta(\mathcal{B});\,\alpha)
+  = I_{\hat\tau}(\hat{a}_1,\hat{b}_1),
+  \qquad
+  \hat\tau = I^{-1}_{1-\alpha}(\hat{a}_0,\hat{b}_0)
+  \label{eq:roll-tpr-at-fpr-beta}
+\end{equation}
+where $I^{-1}_{1-\alpha}(a,b)$ denotes the $(1-\alpha)$-quantile of $\operatorname{Beta}(a,b)$.
+
+\paragraph{Gradient.}
+
+Unlike the Gaussian case, we do not derive closed-form gradients for Beta ROLL.
+The regularised incomplete beta function $I_x(a,b)$ and its inverse are smooth
+functions of all their arguments and are available as differentiable operations in
+PyTorch (\texttt{torch.special.betainc} and \texttt{torch.special.betaincinv}).
+Gradients of \eqref{eq:roll-tpr-at-fpr-beta} with respect to $f_\theta(\mathbf{x}_i)$
+are therefore propagated via automatic differentiation through the estimator
+\eqref{eq:beta-params}, the CDF evaluation, and the quantile inversion.

content/appendixs/appendix_gauss_grad.tex

@@ -1,3 +1,34 @@
+We instantiate the general ROLL framework (\Cref{sec:roll-formulation}) with
+Gaussian-distributed scores.  Each class's scores are modelled as
+$\mathcal{N}(\mu_k, \sigma_k^2)$, fitted by maximum-likelihood estimation.
+
+\paragraph{Forward pass.}
+
+The MLE parameters are:
+\begin{equation}
+  \mu_k = \frac{1}{|\mathcal{B}_k|}\sum_{\mathbf{x}_j \in \mathcal{B}_k} f_\theta(\mathbf{x}_j),
+  \qquad
+  \sigma_k^2 = \frac{1}{|\mathcal{B}_k|}\sum_{\mathbf{x}_j \in \mathcal{B}_k}(f_\theta(\mathbf{x}_j)-\mu_k)^2
+\end{equation}
+
+The CDF and quantile function of $\mathcal{N}(\mu,\sigma^2)$ are:
+\[
+  F(x) = \tfrac{1}{2}\!\left[1 + \operatorname{erf}\!\left(\tfrac{x-\mu}{\sigma\sqrt{2}}\right)\right],
+  \qquad
+  F^{-1}(p) = \mu + \sigma\sqrt{2}\,\operatorname{ierf}(2p-1)
+\]
+where $\operatorname{erf}(x)=\frac{2}{\sqrt{\pi}}\int_0^x e^{-t^2}\,\mathrm{d}t$ is the Gauss
+error function and $\operatorname{ierf}$ its functional inverse.
+Substituting into the general ROLL objective \eqref{eq:roll-tpr-at-fpr}:
+\begin{align}
+  \tau &= \mu_0 + \sigma_0\sqrt{2}\,\operatorname{ierf}(1-2\alpha) \nonumber \\
+  \mathcal{L}_{\text{ROLL-TPR@FPR}}^{\text{GAUSSIAN}}(f_\theta(\mathcal{B});\,\alpha)
+    &= \tfrac{1}{2}\!\left[1 + \operatorname{erf}\!\left(\tfrac{\tau-\mu_1}{\sigma_1\sqrt{2}}\right)\right]
+  \label{eq:roll-tpr-at-fpr-gaussian}
+\end{align}
+
+\paragraph{Gradient derivation.}
+
 We derive the gradient of \eqref{eq:roll-tpr-at-fpr-gaussian} with respect to
 $f_\theta(\mathbf{x}_i)$, instantiating the general structure~\eqref{eq:roll-gradient}
 for the Gaussian case. Throughout, $\tau = \hat{F}_0^{-1}(1-\alpha) =
@@ -65,4 +96,18 @@ Therefore:
   \label{eq:gauss-grad-y0}
 \end{equation}
 
-Substituting \eqref{eq:gauss-grad-y1} and \eqref{eq:gauss-grad-y0} into \eqref{eq:roll-gradient} yields \eqref{eq:gauss-grad-combined}.
+Substituting \eqref{eq:gauss-grad-y1} and \eqref{eq:gauss-grad-y0} into \eqref{eq:roll-gradient}:
+\begin{equation}
+  \frac{\partial \mathcal{L}_{\text{ROLL-TPR@FPR}}^{\text{GAUSSIAN}}}{\partial f_\theta(\mathbf{x}_i)} =
+  \begin{cases}
+    \dfrac{-1}{|\mathcal{B}_1|\,\sigma_1\sqrt{2\pi}}
+    \exp\!\left(-\dfrac{(\tau-\mu_1)^2}{2\sigma_1^2}\right)
+    \!\left(1 + \dfrac{(\tau-\mu_1)(f_\theta(\mathbf{x}_i)-\mu_1)}{\sigma_1^2}\right)
+      & \text{if } y_i = 1 \\[18pt]
+    \dfrac{1}{|\mathcal{B}_0|\,\sigma_1\sqrt{2\pi}}
+    \exp\!\left(-\dfrac{(\tau-\mu_1)^2}{2\sigma_1^2}\right)
+    \!\left(1 + \dfrac{(\tau-\mu_0)(f_\theta(\mathbf{x}_i)-\mu_0)}{\sigma_0^2}\right)
+      & \text{if } y_i = 0
+  \end{cases}
+  \label{eq:gauss-grad-combined}
+\end{equation}

content/introduction/introduction.tex

@@ -1,2 +1,17 @@
-Some references \cite{demaine2007jigsaw, altman1989solving}.
+\section{Problem Statement}
+
+Real-world data is hardly ever perfect. Many real-world datasets suffer a myriad of problems. One such common problem is improper labeling. While we would like our models to learn based off of an exact ground truth, often the labels with which we train our classifiers have errors themselves. Label-noise plagues many datasets for a variety of reasons. In datasets such as CIFAR-10N, a benchmark of natural images across 10 object classes, labels were collected from crowdworker annotators, who are prone to mistakes. In ChestX-ray14, a dataset of medical chest X-rays, labels were given by a natural language processing algorithm, since labelling each image by hand was deemed impractical. Such automatic labeling gave way to relatively high label error rates.
+
+Gradient based learning methods, like those used to train neural networks, have trouble dealing with such datasets. Standard, per-sample losses (BCE, focal-loss), are confused easily by mislabeled points, as they produce a very large gradient which throws off training. Current attempts at noise-robust losses (GCE, MAE) help smooth these large gradients, yet still optimize on a per-sample basis, targeting per-sample accuracy rather than ranking quality --- leaving them misaligned with AUC even when they do reduce noise sensitivity. 
+
+In this field, optimizing directly for our desired metric (AUC) proves to be useful in dealing with noise. Per-sample losses help optimize each sample on their own, while AUC losses optimize the sample ranking amongst the whole. This helps them make smart tradeoffs while training, and accepting errors.
+
+\section{Contribution}
+
+In this work we introduce the \textsc{ROLL} framework for ROC curve related losses and \textsc{ROLL}-AoC, a noise-robust loss that optimizes AUC \emph{directly}. We show that the loss is more robust to label noise than other methods, and how it handles class-imbalance naturally.
+
+\section{Roadmap}
+
+In this work, we will first review AUC optimization, KDE, imbalanced classification, label noise and existing methods for dealing with it in datasets. We then introduce \textsc{ROLL} and \textsc{ROLL}-AoC, formulate it using KDE distribution approximation, and offer derivations for use in gradient-based learning. Then, we show performance on real world data. Finally, we discuss advantages and disadvantages with regards to other methods, as well as offer avenues for future research.
+
 

content/method/method.tex

@@ -155,8 +155,9 @@ score vector in order to obtain a differentiable model of each class's score dis
 \end{equation*}
 %
 Let $\hat{F}_1$ and $\hat{F}_0$ denote the CDFs of $P_{\hat{\phi}_1}$ and $P_{\hat{\phi}_0}$,
-respectively. Specific choices of $\mathcal{F}$ (Gaussian, Beta, KDE) are discussed in
-\Cref{sec:roll-instantiations}.
+respectively. The KDE instantiation is developed in \Cref{sec:roll-instantiations};
+the Gaussian and Beta instantiations, including full derivations, are in
+\Cref{appendix:gauss-grad,appendix:beta-roll}.
 
 Using this notation, at a threshold $\tau$:
 \begin{align}
@@ -253,7 +254,8 @@ Unifying both cases, the general gradient applicable to all ROLL instantiations
   \label{eq:roll-gradient}
 \end{equation}
 The concrete form of each partial derivative depends on the choice of distribution family
-$\mathcal{F}$, and is derived for each instantiation in \Cref{sec:roll-instantiations}.
+$\mathcal{F}$; the KDE case is derived in \Cref{sec:roll-instantiations}, and the
+parametric cases are in \Cref{appendix:gauss-grad,appendix:beta-roll}.
 
 \subsection{ROLL-AOC: AUC Optimization}
 \label{sec:roll-aoc}
@@ -285,8 +287,9 @@ The gradient follows by linearity of the sum:
        {\partial f_\theta(\mathbf{x}_i)}
      \label{eq:roll-aoc-gradient}
 \end{equation}
-The concrete per-threshold gradient used in the sum is derived for each distribution
-family in \Cref{sec:roll-instantiations}.
+The concrete per-threshold gradient for the KDE instantiation is derived in
+\Cref{sec:roll-instantiations}; gradients for the parametric cases are in
+\Cref{appendix:gauss-grad,appendix:beta-roll}.
 
 
 %------------------------------------------------
@@ -294,179 +297,30 @@ family in \Cref{sec:roll-instantiations}.
 \section{Distribution Instantiations}
 \label{sec:roll-instantiations}
 
-% Each subsection below instantiates the general ROLL framework with a specific
-% distributional assumption. Gaussian and Beta are included as parametric
-% special cases; KDE is the primary contribution and receives the fullest treatment.
-
-\subsection{Gaussian ROLL}
-\label{sec:roll-gaussian}
-
-% Assume positive- and negative-class scores are Gaussian distributed. The
-% TPR/FPR integrals reduce to closed-form expressions in terms of the normal
-% CDF, giving a simple and fast-to-compute instantiation of ROLL. Note the
-% limitations: the Gaussian assumption can be violated in practice, motivating
-% the more flexible instantiations below.
-
-A simple method of implementing the ROLL loss is to approximate the distribution
-of the score distributions as gaussian. This method is simple, as there exist
-simple closed-form estimations for the parameters of gaussian distributions, and
-they are differentiable.
-
-While the distributions of scores cannot be gauranteed to follow any form, there are
-some instances where the score distributions might be similar or exactly gaussian.
-For example, when finding linear separability between gaussian distributions, the resulting
-scores will be distributed normally, and this method will fit exactly.
-In ROLL-AOC, the Gaussian instantiation evaluates \eqref{eq:roll-tpr-at-fpr-gaussian}
-at each of the 99 FPR grid points and averages the results.
-
-\subsubsection{Forward}
-\label{sec:roll-gaussian-forward}
-
-% Instantiate \eqref{eq:roll-tpr-at-fpr} under Gaussian score distributions.
-% Each class's scores are modelled as $\mathcal{N}(\mu_k, \sigma_k^2)$, fitted
-% by MLE. The CDF is the standard normal CDF $\Phi$, giving a closed-form loss.
-
-Firstly, we must estimate the parameters to fit the gaussian distributions
-to the true and false populations of the score output.
-
-\begin{equation}
-  \mu_0 = \frac{1}{|\mathcal{B}_0|}\sum_{\mathbf{x}_j \in \mathcal{B}_0}{f_{\theta}(\mathbf{x}_j)}
-  \qquad
-  \mu_1 = \frac{1}{|\mathcal{B}_1|}\sum_{\mathbf{x}_j \in \mathcal{B}_1}{f_{\theta}(\mathbf{x}_j)}
-\end{equation}
-
-\begin{equation}
-  \sigma_0^2 = \frac{1}{|\mathcal{B}_0|}\sum_{\mathbf{x}_j \in \mathcal{B}_0}(f_\theta(\mathbf{x}_j) - \mu_0)^2
-  \qquad
-  \sigma_1^2 = \frac{1}{|\mathcal{B}_1|}\sum_{\mathbf{x}_j \in \mathcal{B}_1}(f_\theta(\mathbf{x}_j) - \mu_1)^2
-\end{equation}
-
-Once these parameters are estimated, we can then define our ROLL objective as required.
-
-The CDF of the normal distribution with mean $\mu$ and standard deviation $\sigma$ is:
-\[
-  F(x) = \frac{1}{2}\left[1 + \text{erf}\!\left(\frac{x - \mu}{\sigma\sqrt{2}}\right)\right]
-\]
-and its inverse (quantile function) is:
-\[
-  F^{-1}(p) = \mu + \sigma\sqrt{2}\,\text{ierf}(2p - 1)
-\]
-where $\operatorname{erf}(x) = \frac{2}{\sqrt{\pi}}\int_0^x e^{-t^2}\,\mathrm{d}t$ is the Gauss
-error function and $\operatorname{ierf}$ denotes its functional inverse, satisfying
-$\operatorname{ierf}(\operatorname{erf}(x)) = x$. Neither has a closed form, but both are
-available in standard numerical libraries (e.g.\ \texttt{torch.erf} and
-\texttt{torch.erfinv}). Plugging into the general ROLL framework, the threshold
-$\tau$ achieving FPR $= \alpha$ and the resulting loss are:
-\begin{align}
-  \tau &= \mu_0 + \sigma_0\sqrt{2}\,\text{ierf}(1 - 2\alpha) \nonumber \\
-  \mathcal{L}_{\text{ROLL-TPR@FPR}}^{\text{GAUSSIAN}}(f_{\theta}(\mathcal{B}) ; \alpha)
-    &= \frac{1}{2}\left[1 + \text{erf}\!\left(\frac{\tau - \mu_1}{\sigma_1\sqrt{2}}\right)\right]
-  \label{eq:roll-tpr-at-fpr-gaussian}
-\end{align}
-
-\subsubsection{Gradient Derivation}
-\label{sec:roll-gaussian-backward}
-
-The full derivation is given in Appendix~\ref{appendix:gauss-grad}.
-Letting $\tau = \hat{F}_0^{-1}(1-\alpha) = \mu_0 + \sigma_0\sqrt{2}\,\operatorname{ierf}(1-2\alpha)$, the gradient is:
-
-\begin{equation}
-  \frac{\partial \mathcal{L}_{\text{ROLL-TPR@FPR}}^{\text{GAUSSIAN}}}{\partial f_\theta(\mathbf{x}_i)} =
-  \begin{cases}
-    \dfrac{-1}{|\mathcal{B}_1|\,\sigma_1\sqrt{2\pi}}
-    \exp\!\left(-\dfrac{(\tau-\mu_1)^2}{2\sigma_1^2}\right)
-    \!\left(1 + \dfrac{(\tau-\mu_1)(f_\theta(\mathbf{x}_i)-\mu_1)}{\sigma_1^2}\right)
-      & \text{if } y_i = 1 \\[18pt]
-    \dfrac{1}{|\mathcal{B}_0|\,\sigma_1\sqrt{2\pi}}
-    \exp\!\left(-\dfrac{(\tau-\mu_1)^2}{2\sigma_1^2}\right)
-    \!\left(1 + \dfrac{(\tau-\mu_0)(f_\theta(\mathbf{x}_i)-\mu_0)}{\sigma_0^2}\right)
-      & \text{if } y_i = 0
-  \end{cases}
-  \label{eq:gauss-grad-combined}
-\end{equation}
-
-\subsection{Beta ROLL}
-\label{sec:roll-beta}
-
-Beta ROLL applies when model scores are constrained to the open interval $(0,1)$,
-as is the case when $f_\theta$ uses a sigmoid output activation.  The Beta distribution
-is supported on $(0,1)$ and its two shape parameters allow it to capture a wide range
-of unimodal and skewed score distributions that a Gaussian cannot represent.
-In ROLL-AOC, Beta ROLL evaluates \eqref{eq:roll-tpr-at-fpr-beta} at each of the 99
-FPR grid points and averages the results, with gradients propagated via automatic
-differentiation.
-
-\subsubsection{Forward}
-\label{sec:roll-beta-forward}
-
-Given score vectors $\mathbf{s}^{(k)} \in (0,1)^{|\mathcal{B}_k|}$ for class $k$, let
-$\mathbf{t}^{(k)} = \mathbf{1} - \mathbf{s}^{(k)}$ denote the complementary scores.
-For any vector $\mathbf{v}$, write $\bar{v} = \frac{1}{|\mathbf{v}|}\sum_i v_i$.
-We estimate the shape parameters using the closed-form log-moment estimator
-of~\cite{chen2022beta}: define for each class $k$,
-
-\begin{equation}
-  D_k = \overline{s^{(k)}\!\ln s^{(k)}} - \bar{s}^{(k)}\cdot\overline{\ln s^{(k)}}
-      + \overline{t^{(k)}\!\ln t^{(k)}} - \bar{t}^{(k)}\cdot\overline{\ln t^{(k)}}
-  \label{eq:beta-denom}
-\end{equation}
-
-\begin{equation}
-  \hat{a}_k = \frac{\bar{s}^{(k)}}{D_k}, \qquad \hat{b}_k = \frac{\bar{t}^{(k)}}{D_k}
-  \label{eq:beta-params}
-\end{equation}
-
-These estimators follow from two simultaneous moment conditions:
-$\mathbb{E}[X] = a/(a+b)$ and
-$\operatorname{Cov}(X,\ln X) + \operatorname{Cov}(Y,\ln Y) = 1/(a+b)$,
-where $Y=1-X\sim\operatorname{Beta}(b,a)$.
-Replacing expectations by sample means and solving in closed form yields~\eqref{eq:beta-params}.
-
-The CDF of $\operatorname{Beta}(\hat{a}_k,\hat{b}_k)$ at $x\in(0,1)$ is the
-regularised incomplete beta function:
-\begin{equation}
-  \hat{F}_k(x) = I_x(\hat{a}_k,\hat{b}_k)
-               = \frac{B(x;\,\hat{a}_k,\hat{b}_k)}{B(\hat{a}_k,\hat{b}_k)},
-  \qquad B(x;\,a,b) = \int_0^x u^{a-1}(1-u)^{b-1}\,\mathrm{d}u
-  \label{eq:beta-cdf}
-\end{equation}
-
-Substituting into the general ROLL framework \eqref{eq:roll-tpr-at-fpr}:
-\begin{equation}
-  \mathcal{L}_{\text{ROLL-TPR@FPR}}^{\text{BETA}}(f_\theta(\mathcal{B});\,\alpha)
-  = I_{\hat\tau}(\hat{a}_1,\hat{b}_1),
-  \qquad
-  \hat\tau = I^{-1}_{1-\alpha}(\hat{a}_0,\hat{b}_0)
-  \label{eq:roll-tpr-at-fpr-beta}
-\end{equation}
-where $I^{-1}_{1-\alpha}(a,b)$ denotes the $(1-\alpha)$-quantile of $\operatorname{Beta}(a,b)$.
-
-\subsubsection{Gradient Derivation}
-\label{sec:roll-beta-backward}
-
-Unlike the Gaussian case, we do not derive closed-form gradients for Beta ROLL.
-The regularised incomplete beta function $I_x(a,b)$ and its inverse are smooth
-functions of all their arguments and are available as differentiable operations in
-PyTorch (\texttt{torch.special.betainc} and \texttt{torch.special.betaincinv}).
-Gradients of \eqref{eq:roll-tpr-at-fpr-beta} with respect to $f_\theta(\mathbf{x}_i)$
-are therefore propagated via automatic differentiation through the estimator
-\eqref{eq:beta-params}, the CDF evaluation, and the quantile inversion.
+The abstract ROLL framework (\Cref{sec:roll-formulation}) accommodates any distribution
+family $\mathcal{F}$ whose CDF and quantile function are differentiable with respect to
+the model scores.  As part of this work, we instantiated the framework for two parametric
+families: a Gaussian model, fitted by maximum likelihood, and a Beta model, fitted by a
+closed-form log-moment estimator~\cite{chen2022beta}.  Both admit fast parameter estimation
+and differentiable CDFs; for the Gaussian case we derive analytic gradients in full, and
+for the Beta case gradients propagate via automatic differentiation.  Complete derivations
+are given in \Cref{appendix:gauss-grad,appendix:beta-roll}.
 
+Parametric families, however, impose distributional assumptions on the score output that
+no trained model can be guaranteed to satisfy.  The primary instantiation in this thesis
+is therefore non-parametric: Kernel Density Estimation (KDE) places no assumption on the
+shape of the score distribution and adapts to whatever the model produces at each training
+step.  The cost is a numerical CDF inversion and a custom backward pass; the gain is a
+faithful surrogate for the ROC curve regardless of how scores evolve during training.
 
 \subsection{KDE ROLL}
 \label{sec:roll-kde}
 
-% KDE background: see \Cref{sec:kde-background} in \Cref{chapter:related_work}.
-% The primary instantiation: non-parametric, no assumption on score shape.
-% Discuss the trade-off between flexibility and computational cost, and how
-% the bandwidth (see \Cref{sec:kde-bandwidth}) controls bias-variance.
-
-Unfortunately, the score output of trained models remains unpredictable. A solution for this is to use a Kernel Density Estimation (KDE) for probability
-estimation. This benefits from being far more adaptable to real-world distributions of outputs of our trained models. The tradeoff of course is computational
-complexity - computing the gradient derivation in such a case is non-trivial.
-KDE ROLL is the primary instantiation used in ROLL-AOC: its non-parametric flexibility
-makes it the most accurate estimator of the ROC curve when score distributions are
-unknown, at the cost of evaluating the custom backward pass 99 times per training step.
+KDE estimates the score distribution directly from the observed batch scores, placing no
+assumption on their shape.  It is the primary instantiation of ROLL used throughout this
+thesis: its non-parametric flexibility makes it the most faithful surrogate for the ROC
+curve when score distributions are unknown, at the cost of a numerical CDF inversion and
+a custom backward pass evaluated 99 times per training step under ROLL-AOC.
 
 \subsubsection{Forward}
 \label{sec:roll-kde-forward}

content/related_work/related_work.tex

@@ -74,7 +74,7 @@ The noise-robustness properties of ranking-based objectives are discussed in \Cr
 
 While AUC as a metric for model performance has existed for a while, methods to optimize for it directly are relatively recent. At it's core, it is a non-trivial problem to solve. Existing efforts to directly optimize AUC have been based on the Wilcoxon-Mann-Whitney interpretation of AUC. However, directly calculating the afformentioned probability of a random positive score behing higher than a negative one introduces two complications. The first is that one must iterate over $n^2$ pairs to cover every posible pair. Additionally, the pairwise function one must evaluate ( $\mathbb{P}(f(x_1) > f(x_0))$ ) is a non-contiguous stepwise function, and is therefor on-differentiable.
 
-The non-contiguous stepwise nature of the evaluated pairwise function is generally fixed by using a surrogate — either sigmoid-like or hinge functions are applied to the score difference~\cite{cortes2003auc}.
+The non-contiguous stepwise nature of the evaluated pairwise function is generally fixed by using a surrogate, either sigmoid-like or hinge functions are applied to the score difference~\cite{cortes2003auc}.
 \citet{yuan2021auc} proposed a compositional stochastic gradient framework that reformulates AUC maximization as a min-max saddle-point problem, enabling mini-batch training without maintaining all pairwise scores in memory. Their PESG optimizer and the LibAUC library are the primary practical tools for deep AUC maximization and serve as the main AUC baseline in this thesis.
 
 % ============================================================
@@ -272,7 +272,8 @@ on the rare class during training, as it contributes less to the loss metric, an
 generalize on the rare class poorly.
 
 Dealing with this depends on the loss used. For most losses, for which the loss for each sample is calculated seperately,
-is to simply weight the losses differently, by multiplying the loss for the rare class
+the solution is to simply weight the losses for each class differently,
+by multiplying the loss for the rare class
 so that it contributes equally. Other approaches using pairwise losses, such as those implemented in LibAUC \cite{yuan2021auc}, naturally weight positive and negative contributions to learning through their implementation.
 
 % ============================================================

main.tex

@@ -192,18 +192,18 @@
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %Title and names for the article
-\newcommand{\thesistitle}{\textbf{Thesis Title}}
-\newcommand{\thesistitlehe}{\textbf{כותרת בעברית של התזה}}
+\newcommand{\thesistitle}{\textbf{Noise-Robust Gradient-Based Learning Utilizing Rate Optimized Likelyhood Loss}}
+\newcommand{\thesistitlehe}{\textbf{למידה עמידה לרעש מבוססת גרדיאנט תוך שימוש בפונקציית הפסד קירוב-הסתברות}}
 
-\newcommand{\thesisauthorname}{\textbf{Author’s name}}
-\newcommand{\thesisauthornamehe}{\textbf{שם המחבר}}
+\newcommand{\thesisauthorname}{\textbf{Aner Zakobar}}
+\newcommand{\thesisauthornamehe}{\textbf{ענר זקובר}}
 
-\newcommand{\thesissupervisername}{\textbf{Supervisor's name}}
-\newcommand{\thesissupervisernamehe}{\textbf{שם המנחה}}
+\newcommand{\thesissupervisername}{\textbf{Prof. Lior Rokach, Prof. Aryeh Kontorovich}}
+\newcommand{\thesissupervisernamehe}{\textbf{פרופ. ליאור רוקח, פרופ. אריה קונטורוביץ}}
 
-\newcommand{\thesismonth}{\textbf{Month}}
-\newcommand{\thesismonthhe}{\textbf{חודש}}
-\newcommand{\thesisyear}{\textbf{Year}}
+\newcommand{\thesismonth}{\textbf{September}}
+\newcommand{\thesismonthhe}{\textbf{ספטמבר}}
+\newcommand{\thesisyear}{\textbf{2026}}
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
 \begin{document}
@@ -261,34 +261,39 @@
 \addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח א}{\quad\qquadדרישת תאימות  כלשהי}}{{\beginL\thepage\endL}}}
 \input{content/appendixs/appendix_a}
 
-\chapter{Gaussian ROLL: Gradient Derivation}
+\chapter{Gaussian ROLL: Forward Pass and Gradient Derivation}
 \label{appendix:gauss-grad}
 \addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ב}{\quad\qquadנגזרת גאוסיאנית של ROLL}}{{\beginL\thepage\endL}}}
 \input{content/appendixs/appendix_gauss_grad}
 
+\chapter{Beta ROLL: Forward Pass and Gradient}
+\label{appendix:beta-roll}
+\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ג}{\quad\qquadBeta ROLL: אמידה ונגזרת}}{{\beginL\thepage\endL}}}
+\input{content/appendixs/appendix_beta_roll}
+
 \chapter{KDE ROLL: Gradient Derivation}
 \label{appendix:kde-grad}
-\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ג}{\quad\qquadנגזרת KDE של ROLL}}{{\beginL\thepage\endL}}}
+\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ד}{\quad\qquadנגזרת KDE של ROLL}}{{\beginL\thepage\endL}}}
 \input{content/appendixs/appendix_kde_grad}
 
 \chapter{KDE ROLL: Sigmoid Kernel — Explicit Equations}
 \label{appendix:kde-sigmoid}
-\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ד}{\quad\qquadKDE ROLL: ניסוח מפורש עם גרעין סיגמואיד}}{{\beginL\thepage\endL}}}
+\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ה}{\quad\qquadKDE ROLL: ניסוח מפורש עם גרעין סיגמואיד}}{{\beginL\thepage\endL}}}
 \input{content/appendixs/appendix_kde_sigmoid}
 
 \chapter{KDE ROLL: Bisection for CDF Inversion}
 \label{appendix:kde-bisection}
-\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ה}{\quad\qquadKDE ROLL: היפוך CDF בשיטת חצייה}}{{\beginL\thepage\endL}}}
+\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ו}{\quad\qquadKDE ROLL: היפוך CDF בשיטת חצייה}}{{\beginL\thepage\endL}}}
 \input{content/appendixs/appendix_kde_bisection}
 
 \chapter{KDE ROLL: Newton--Raphson Inversion (Superseded)}
 \label{appendix:kde-nr}
-\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ו}{\quad\qquadKDE ROLL: היפוך ניוטון-רפסון}}{{\beginL\thepage\endL}}}
+\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ז}{\quad\qquadKDE ROLL: היפוך ניוטון-רפסון}}{{\beginL\thepage\endL}}}
 \input{content/appendixs/appendix_kde_nr}
 
 \chapter{ROLL Loss: Implementation Listing}
 \label{appendix:roll-code}
-\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ז}{\quad\qquadROLL: קוד מימוש}}{{\beginL\thepage\endL}}}
+\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{נספח ח}{\quad\qquadROLL: קוד מימוש}}{{\beginL\thepage\endL}}}
 \input{content/appendixs/appendix_roll_code}
 
 \end{appendices}
@@ -305,4 +310,4 @@
 \include{Main_pages/abstract_heb}
 \include{Main_pages/cover_heb}
 
-\end{document}
+\end{document}