aner/thesis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Changes adding things and dir locals

Browse Source
This commit is contained in:
Aner Zakobar
2026-08-20 10:18:30 +03:00
parent 1a8a31c158
commit 86291e948e
4 changed files with 231 additions and 132 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/related_work/related_work.tex
+158 -125
View File
@@ -113,6 +113,12 @@ TODO finish this section
% Key limitation shared by all: they optimize the \emph{whole-curve} AUC, not a specific
% operating point on the ROC --- motivating the next section.
\subsection{AUC and ROC definitions}
% TODO add formalizations
% Formalize ROC and integration into AUC
A Receiver Operating Characteristic (ROC) is a graph that represents the performance of a classifier.
The classifier in question is evaluated on binary-labeled data. The labeled data is ordered based off of the
scores given by the classifier. Given any numerical threshold, it is then easy to see which labeled data points fall above and below.
@@ -122,14 +128,21 @@ For each threshold, the true positive rate and the false positive rate is given.
The AUC is a number in the range $[0, 1]$ representing the total area under the curve. It is calculated by integrating over all TPRs and taking the resulting average FPRs. A result of $1$ represents a perfect classifier,
while a result of $0$ represents a classifier with a perfect opposite result. A result of $0.5$ represents a classifier that found no separation whatsoever.
\subsection{AUC as a metric of model performance}
The AUC is a direct representation of the performance of a classifier, as opposed to a loss. There are instances where the loss may be minimized, yet seperation might be minimal. These two metrics might correlate, but do not necessarily correspond.
An example of this is a result for which all true samples receive a score of $0.51$ and all false samples receie a score of $0.49$. Using many losses, including BCE, this would result in a relatively high loss value. On this score however, the AUC is $1$. The model perfectly seperates the samples, so further training is not necessary.
Direct AUC optimization can be ieal with optimizing for imbalanced datasets. With most losses such as BCE or focal, the loss is calculated per-sample and then summed (or averaged). In order to properly train for both true and false populations, the scores are weighted, so that equal weight falls on both the true and the false populations. With direct AUC optimization, the imbalance does not affect the calculation, and the reweighting can be forgone.
Direct AUC optimization can be ideal with optimizing for imbalanced datasets. With most losses such as BCE or focal, the loss is calculated per-sample and then summed (or averaged). In order to properly train for both true and false populations, the scores are weighted, so that equal weight falls on both the true and the false populations. With direct AUC optimization, the imbalance does not affect the calculation, and the reweighting can be forgone.
Additionally, direct AUC methods are more resilient to label noise, as opposed to classical methods. Due to inseperability, correctly classifying a sample might require a model to incorrectly classify similar or even identical samples that were mislabeled. This is only worth it for the model in training if the total loss of the correctly labeled samples outweighs the total loss of the incorrectly labeled samples. For most methods, the weight per class is given by the class imbalance in the dataset, but this doesn't take into account the label noise in any way. Optimizing for AUC specifically would eliminate this risk.
\subsection{AUC optimization in literature}
While AUC as a metric for model performance has existed for a while, methods to optimize for it directly are relatively recent.
%TODO add references
% ============================================================
% PLACEMENT: Section 2.3, after ROC/AUC section above and
% before the KDE background section. Narrows from whole-curve
@@ -190,7 +203,42 @@ TODO write this section.
% block inside a loss function. Forward pointer: \Cref{sec:roll-kde} exploits this to
% construct ROLL's differentiable threshold estimate from the estimated score distributions.
TODO write this section.
When describing the source distribution for samples, rarely do they conform to some parametric distribution.
Some data, such as adult heights, follows a Gaussian-like distribution; the occurrence of radioactive decay
events or car arrivals at an intersection follows a Poisson distribution. For arbitrary real-world data,
however, no such assumption can be made safely.
This motivates Kernel Density Estimation (KDE). Rather than assuming a parametric form, KDE uses the
empirical sample itself as the model: each observed point $x_i$ contributes a smooth bump --- a
\emph{kernel} --- to the estimated density. The assumption is that nearby points are plausibly likely,
so probability mass is spread locally around each sample. Summing all kernels and normalizing yields a
smooth PDF that mirrors the shape of the underlying distribution without committing to a parametric family.
Formally, let $\{x_i\}_{i=1}^n$ be the observed samples, $K$ a kernel function satisfying
$\int_{-\infty}^{\infty} K(u)\,du = 1$, and $h > 0$ the \emph{bandwidth} controlling the spread of
each kernel. Setting the correct bandwidth is important - too low a bandwidth parameter and the KDE becomes too jagged, and the probability anywhere there was no candidate approaches zero, avoiding generalization. Too high a bandwidth parameter, on the other hand, smooths out the distribution entirely, and the entire distribution starts to approach that of the kernel itself, losing the specificity that the candidates bring.
The KDE density estimator is:
\[
\hat{p}(x) = \frac{1}{nh} \sum_{i=1}^{n} K\!\left(\frac{x - x_i}{h}\right)
\]
Each term $\frac{1}{h}K\!\bigl(\frac{x-x_i}{h}\bigr)$ is a kernel centered at $x_i$; the
$1/h$ factor compensates for the change of variables and ensures the full sum integrates to 1.
Valid kernels must be non-negative, symmetric around zero, and normalized. These differ in smoothness, tail behavior, and computational cost. A particularly useful choice is
the \emph{logistic kernel}:
\[
K(u) = \frac{e^{-u}}{(1 + e^{-u})^2}
\]
which is smooth, bell-shaped, and centered at zero. Its key property is that its CDF has a closed form,
which is the sigmoid function:
\[
\int_{-\infty}^{x} K(u)\,du \;=\; \frac{1}{1+e^{-x}} \;=:\; \sigma(x)
\]
This gives the KDE cumulative distribution function a closed form as well:
\[
\hat{F}(\tau) = \frac{1}{n}\sum_{i=1}^{n} \sigma\!\left(\frac{\tau - x_i}{h}\right)
\]
% ============================================================
@@ -307,7 +355,114 @@ TODO write this section.
% conferring noise robustness without explicit correction.
% Empirical evidence deferred to the Experiments chapter.
TODO write this section.
While collecting data with correct labels should be a priority, in some cases it is impossible.
Many data collection methods lead to imperfect labeling, resulting in a muddy ground truth.
This could be due to a number of factors. The labels might be based on human labeling, which is error prone.
The labeling might be true, yet the signal might be noisy or wrong, creating a mismatch between signal and label.
The labeling could be a resource-intensive process, and so it might only be done for a portion of the data.
And in some cases, the labeling is done based off a different learned model, which too could be faulty.
\subsection{Label noise formalization}
Given a dataset $ \mathcal{D} = \{(x_i, y_i)\} $, we would define the noisy version as $ \tilde{\mathcal{D}} = \{(x_i, \tilde{y_i})\} $. The noise $N_i$ is defined as:
\[
N_i =
\begin{cases}
0 & \text{if } y_i = \tilde{y_i} \\
1 & \text{otherwise}
\end{cases}
\]
Label noise is usually categorized as being symmetric, asymmetric or instance dependent.
For instance dependent noise, $\mathbb{P}(N_i = 1)$ depends on $x_i$. That is, the probability that a label is flipped depends on $x_i$. For the other two types, the probability of the label being flipped depends only on the original label itself.
For symmetric noise, $\mathbb{P}(N_i = 1 | y_i = 1) = \mathbb{P}(N_i = 1 | y_i = 0)$. The probability of the label being flipped does not depend on the original label. This is unlike asymmetric noise, for which the probability of the label flipping does depend on the original label.
\subsection{Existing approaches}
Prior work has been done in order to make noise-robust learning processes. These can be categorized into three main approaches.
\subsubsection{Label meta-training and adjustment}
Under this approach, the training process is adjusted by modelling label noise explicitly.
\citet{patrini2017making} introduced forward and backward loss correction: by estimating a
noise transition matrix $T$, where $T_{ij} = P(\tilde{y}=j \mid y=i)$, the observed noisy
loss can be corrected to recover the clean-label risk.
Estimating $T$ requires identifying high-confidence \emph{anchor points} for each class from
a pre-trained noisy model --- a two-phase procedure that may fail under high or asymmetric
noise rates.
\citet{ren2018learning} sidestep transition matrix estimation by using a small clean
validation set and meta-gradient descent to assign per-sample loss weights at each training
step, adapting dynamically to whatever noise is present without assuming a fixed noise
structure.
Both approaches require either prior knowledge of the noise model or access to a clean
held-out set, which is not always available in practice.
\subsubsection{Sample selection}
Rather than modifying the loss, sample selection methods filter likely-noisy examples during
training.
The approach is grounded in the \emph{memorization effect} documented by
\citet{arpit2017memorization}: deep networks learn clean-label patterns first and memorize
noisy labels only later in training.
This means that at early training stages, samples with small loss are more likely to be
correctly labelled and can be preferentially retained.
\citet{han2018coteaching} exploit this with Co-teaching: two networks train in parallel,
each selecting its own small-loss samples and passing them to the other network for the
parameter update, preventing both from converging to the same memorized errors.
\citet{li2020dividemix} extend the idea by fitting a per-class Gaussian mixture model to
the loss distribution at each epoch to separate clean from noisy samples, then treating the
noisy partition as unlabeled data within a semi-supervised learning framework.
These methods can achieve strong performance but substantially increase training complexity,
requiring multiple models or multi-phase procedures.
\subsubsection{Robust loss functions}
A third line of work designs loss functions that are intrinsically resistant to label noise,
requiring no changes to the training procedure beyond swapping the loss term.
\citet{ghosh2017robust} established a sufficient condition for noise tolerance: a loss
$\ell$ tolerates uniform symmetric noise if and only if
\[
\sum_{k=1}^{K} \ell(f(\mathbf{x}), k) = C \quad \text{for all } \mathbf{x},\, f
\]
for some constant $C$ independent of $f(\mathbf{x})$.
Intuitively, if the total loss over all class assignments is constant, randomly flipping
labels cannot shift the expected risk minimizer.
Mean Absolute Error (MAE) satisfies this condition --- one can verify that
$\sum_k (1 - f_k(\mathbf{x})) = K - 1$ for a $K$-class softmax output ---
whereas cross-entropy does not.
However, MAE converges slowly in practice because its gradient has constant magnitude
regardless of prediction confidence, making it insensitive to easy examples that should
receive small updates.
\citet{zhang2018generalized} proposed the Generalised Cross Entropy (GCE) loss, defined via
the $L_q$ family:
\[
\ell_q(f(\mathbf{x}), y) = \frac{1 - f_y(\mathbf{x})^q}{q}, \quad q \in (0,\,1]
\]
where $f_y(\mathbf{x})$ is the model's predicted probability for the true class $y$.
As $q \to 0$ the loss recovers cross-entropy (via L'H\^{o}pital's rule); at $q = 1$ it
reduces to MAE, which satisfies the symmetry condition.
Intermediate values therefore trade noise tolerance for faster convergence.
We evaluate GCE at $q = 0.7$ as a baseline in the Experiments chapter.
\citet{zhou2021asymmetric} extend the theoretical framework to asymmetric noise, introducing
the \emph{asymmetry ratio} to quantify how noise tolerance degrades when flip rates differ
between classes.
\citet{feng2020can} confirm that standard cross-entropy cannot be made noise-tolerant
through simple modification, establishing robust losses as a genuinely distinct family
rather than a patch on existing objectives.
All three families above treat noise as a quantity to be explicitly corrected --- by
adjusting labels, filtering samples, or redesigning the loss.
ROLL takes a different route: its distributional threshold objective aggregates over the
full score distribution of each class, so a small fraction of mislabelled samples shifts
the estimated density slightly but cannot dominate the threshold.
This structural robustness arises from the distributional formulation itself, without any
explicit noise-handling mechanism; empirical evidence is provided in the Experiments
chapter.
% ============================================================
% PLACEMENT: Section 2.6 (was 2.5), near the END of Ch. 2,
@@ -334,125 +489,3 @@ TODO write this section.
TODO write this section.
%------------------------------------------------
\section{topic a}
\label{sec:related_work:jigsaw_puzzles}
\addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{נושא א}}
To be continued.
\section{topic b}
\label{sec:related_work:relaxation_labeling}
\addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{נושא ב}}
To be continued.
\subsection{sub topic b.1}
\label{subsec:formulation_as_rl:rationale:type_2}
\addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{תת נושא ב1}}
To be continued.
\begin{figure}[H]
\centering
\begin{subfigure}[b]{0.3\textwidth}
\begin{tikzpicture}
\node[anchor=south west, inner sep=0] at (0,0) {\includegraphics[width=\textwidth]{content/related_work/images/2x2_puzzle_grid.png}};
\draw[step=0.5\textwidth] (0,0) grid (\textwidth,\textwidth);
\node[font=\large] at (0.25\textwidth,0.75\textwidth) {(1,1)};
\node[font=\large] at (0.75\textwidth,0.75\textwidth) {(1,2)};
\node[font=\large] at (0.25\textwidth,0.25\textwidth) {(2,1)};
\node[font=\large] at (0.75\textwidth,0.25\textwidth) {(2,2)};
\end{tikzpicture}
\caption{}
\label{fig:type_1_goal_and_labeling:dimensions}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\begin{tikzpicture}
\node[anchor=south west, inner sep=0] at (0,0) {\includegraphics[width=\textwidth]{content/related_work/images/2x2puzzle_type_1.png}};
\draw[step=0.5\textwidth] (0,0) grid (\textwidth,\textwidth);
\node[font=\large, color=red] at (0.25\textwidth,0.75\textwidth) {Piece 1};
\node[font=\large, color=red] at (0.75\textwidth,0.75\textwidth) {Piece 2};
\node[font=\large, color=red] at (0.25\textwidth,0.25\textwidth) {Piece 3};
\node[font=\large, color=red] at (0.75\textwidth,0.25\textwidth) {Piece 4};
\end{tikzpicture}
\caption{}
\label{fig:type_1_goal_and_labeling:pieces}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\begin{tikzpicture}
\node[anchor=south west, inner sep=0] at (0,0) {\includegraphics[width=\textwidth]{content/related_work/images/2x2puzzle_solution.png}};
\draw[step=0.5\textwidth] (0,0) grid (\textwidth,\textwidth);
\node[font=\large, color=red] at (0.25\textwidth,0.75\textwidth) {Piece 3};
\node[font=\large, color=red] at (0.75\textwidth,0.75\textwidth) {Piece 2};
\node[font=\large, color=red] at (0.25\textwidth,0.25\textwidth) {Piece 4};
\node[font=\large, color=red] at (0.75\textwidth,0.25\textwidth) {Piece 1};
\end{tikzpicture}
\caption{}
\label{fig:type_1_goal_and_labeling:solution}
\end{subfigure}
\vfill
\begin{subfigure}[b]{1\textwidth}
\centering
\begin{tikzpicture}
\def\scaletitles{0.88}
\def\minimumEntrySize{0.95cm}
\matrix[matrix of nodes,
nodes={draw, align=center, minimum size=\minimumEntrySize},
row 1/.style={nodes={draw=none, gray, font=\footnotesize, scale=\scaletitles}},
column 1/.style={nodes={draw=none, gray, font=\footnotesize,
scale=\scaletitles}}]
{
\node{}; &
\node{(1,1)}; &
\node{(1,2)}; &
\node{(2,1)}; &
\node{(2,2)};\\
%----------%
\node{Piece 1}; &
\node{0}; & \node{0}; & \node{0}; & \node[text=blue]{1};\\
%----------%
\node{Piece 2}; &
\node{0}; & \node[text=blue]{1}; & \node{0}; & \node{0};\\
%----------%
\node{Piece 3}; &
\node[text=blue]{1}; & \node{0}; & \node{0}; & \node{0};\\
%----------%
\node{Piece 4}; &
\node{0}; & \node{0}; & \node[text=blue]{1}; & \node{0};\\
};
\end{tikzpicture}
\caption{}
\label{fig:type_1_goal_and_labeling:labeling}
\end{subfigure}
\caption[fig A - Example for fig]{Some example}
\label{fig:type_1_goal_and_labeling}
\end{figure}
\begin{table}[H]
\centering
\begin{tabular}{ |c|c|c|c|c|c| }
\hline
\multicolumn{1}{|c|}{Puzzle Type} & Direct & Neighbor & Perfect & Occupied & Feasible \\
\hline
Type 1 & {0\%} & {0.1\%} & 0 & {1\%} & 0 \\
\hline
Type 2 & {0.001\%} & {0.1\%} & 0 & {1.2\%} & 0 \\
\hline
\end{tabular}
\caption[Table A - Example for table]{Some example}
\label{table:plain_rl_results}
\end{table}