119 lines
7.7 KiB
TeX
119 lines
7.7 KiB
TeX
This appendix gives a Neyman-Pearson-theoretic account of why order-dependent losses (objectives that depend on the ranking of scores rather than on comparing each score
|
|
to a fixed threshold) avoid a real limitation of class-weighted
|
|
per-sample losses. \Cref{sec:np-class-weighting} shows that a weighted per-sample loss
|
|
commits to a Neyman-Pearson threshold $\eta$ chosen indirectly through the class
|
|
weights.
|
|
\Cref{sec:np-noise-invariance} shows that under label noise this implicit $\eta$ is
|
|
further, unpredictably distorted. Neither limitation is specific to \textsc{Roll}: they
|
|
apply to any per-sample-weighted loss, and the way around them (targeting an operating
|
|
point or optimizing AUC directly, without ever forming a class weight or comparing to a
|
|
fixed $\eta$) applies equally to \textsc{Roll}'s TPR@FPR and ROLL-AOC objectives and to
|
|
other AUC-optimizing methods such as \textsc{LibAUC}~\cite{yuan2021auc}.
|
|
|
|
\section{Class Weighting as an Implicit Choice of $\eta$}
|
|
\label{sec:np-class-weighting}
|
|
|
|
Recall from \Cref{sec:neyman-pearson} that the Neyman-Pearson lemma characterizes the
|
|
optimal test at a fixed false-positive rate as thresholding the likelihood ratio
|
|
$p(\mathbf{x}\mid y{=}1)/p(\mathbf{x}\mid y{=}0)$ against some constant $\eta$: a single
|
|
threshold, fixed once for the whole test, the same for every $\mathbf{x}$. We will not
|
|
need $\eta$'s exact value below, only that it exists, that it is one constant shared by
|
|
every sample, and that a weighted per-sample loss determines it indirectly through the
|
|
class weights.
|
|
|
|
\Cref{sec:imbalanced-tpr-fpr} noted that per-sample losses typically correct for class
|
|
imbalance by multiplying the loss on one class by a weight. This practice has a direct
|
|
reading in Neyman-Pearson terms: for a weighted per-sample loss, minimizing the pointwise
|
|
loss and thresholding the result is equivalent to thresholding the posterior
|
|
odds,\footnote{For weighted cross-entropy with weights $w_1,w_0$, minimizing
|
|
$w_1 P(y{=}1\mid\mathbf{x})(-\log q) + w_0 P(y{=}0\mid\mathbf{x})(-\log(1-q))$ over the
|
|
predicted probability $q$ gives $q^\star = w_1 P(y{=}1\mid\mathbf{x}) \big/
|
|
\bigl[w_1 P(y{=}1\mid\mathbf{x}) + w_0 P(y{=}0\mid\mathbf{x})\bigr]$. Thresholding
|
|
$q^\star > \tfrac12$ is equivalent to $w_1 P(y{=}1\mid\mathbf{x}) > w_0 P(y{=}0\mid\mathbf{x})$,
|
|
i.e. to the threshold on $O(\mathbf{x})$ stated below.} $O(\mathbf{x}) = P(y{=}1\mid\mathbf{x})/P(y{=}0\mid\mathbf{x})$,
|
|
against a constant set by the weight ratio:
|
|
\[
|
|
O(\mathbf{x}) > \eta, \qquad \eta = \frac{w_0}{w_1}.
|
|
\]
|
|
This exact value of $\eta$ follows from this particular (weighted cross-entropy) loss;
|
|
other weighted per-sample losses would fix it slightly differently. What matters below is
|
|
not the exact value but the qualitative fact common to all of them: $\eta$ is one constant,
|
|
the same for every sample, that moves with the chosen weight ratio. Choosing per-class
|
|
weights is therefore, knowingly or not, equivalent to choosing $\eta$ indirectly, and
|
|
without prior knowledge of the actual separability of the data.
|
|
|
|
Concretely, for any point $\mathbf{x}$, this means the classifier ends up treating
|
|
$\mathbf{x}$ as positive or negative purely according to which side of $\eta$ its own
|
|
posterior odds $O(\mathbf{x})$ falls on: samples with $O(\mathbf{x}) > \eta$ are pushed
|
|
toward the positive class, samples with $O(\mathbf{x}) < \eta$ toward the negative class,
|
|
regardless of how separable $\mathbf{x}$ actually is from the other class. Two samples with
|
|
nearly identical posterior odds straddling $\eta$ can therefore end up on opposite sides of
|
|
the decision rule purely because of where $\eta$ happens to land, and since $\eta$ is
|
|
fixed entirely by the chosen weight ratio, it is the weight choice, not any property of the
|
|
data, that decides the fate of every such borderline sample. This is the central difference
|
|
from an order-dependent loss: a per-sample loss must commit to some $\eta$ before training
|
|
even begins, and every sample's outcome is filtered through that fixed, weight-determined
|
|
cutoff.
|
|
|
|
Ordering-based objectives, such as our ROLL instantiations, avoid this altogether: rather
|
|
than fixing $\eta$ via class weights, they optimize for AUC or a target operating point
|
|
directly, and whatever $\eta$ this implies is discovered from the fitted score
|
|
distributions during training. No class weight, and no assumption linking class counts to
|
|
$\eta$, is needed. The same is true of any objective that optimizes AUC or a fixed
|
|
operating point directly, including \textsc{LibAUC}: none of these ever form a
|
|
class-weight ratio or compare a probability to a fixed $\eta$ in the first place.
|
|
|
|
\section{Class Weighting Under Label Noise: An Implicit, Distorted $\eta$}
|
|
\label{sec:np-noise-invariance}
|
|
|
|
\Cref{sec:np-class-weighting} showed that a weighted per-sample loss trained on clean
|
|
labels implicitly targets a threshold $\eta$ on the true posterior odds
|
|
$O(\mathbf{x}) = P(y{=}1\mid\mathbf{x})/P(y{=}0\mid\mathbf{x})$, set by the weight ratio.
|
|
In practice, however, training data is rarely clean. This section shows that under label
|
|
noise, the threshold actually reached is not the one chosen, regardless of how $\eta$
|
|
itself is defined.
|
|
|
|
Suppose labels are corrupted by class-conditional noise, the same model used in
|
|
\Cref{sec:noisy-labels} and in this thesis's noise experiments: each label is flipped with
|
|
probability $\rho \in [0,\tfrac12)$ depending only on its true class, independent of
|
|
$\mathbf{x}$. A model trained on the resulting noisy labels $\tilde y$ estimates the noisy
|
|
posterior $P(\tilde y{=}1\mid\mathbf{x})$, not the true one, so a weight ratio $w = w_0/w_1$
|
|
now thresholds $P(\tilde y{=}1\mid\mathbf{x})/P(\tilde y{=}0\mid\mathbf{x})$, not
|
|
$O(\mathbf{x})$. These two quantities are related by a fixed function of $\rho$
|
|
alone,\footnote{Since the noise depends only on $y$, $P(\tilde y{=}1\mid\mathbf{x}) =
|
|
(1-\rho)P(y{=}1\mid\mathbf{x}) + \rho P(y{=}0\mid\mathbf{x})$, and similarly
|
|
$P(\tilde y{=}0\mid\mathbf{x}) = \rho P(y{=}1\mid\mathbf{x}) + (1-\rho)P(y{=}0\mid\mathbf{x})$.
|
|
Dividing both by $P(y{=}0\mid\mathbf{x})$ and taking the ratio gives
|
|
$P(\tilde y{=}1\mid\mathbf{x})/P(\tilde y{=}0\mid\mathbf{x}) = h(O(\mathbf{x}))$ with
|
|
$h(O) = [\rho + O(1-\rho)] / [(1-\rho) + O\rho]$, and $h'(O) = (1-2\rho) \big/
|
|
[(1-\rho)+O\rho]^2 > 0$ for $\rho < \tfrac12$, independent of $O$.} namely
|
|
\[
|
|
\frac{P(\tilde y{=}1\mid\mathbf{x})}{P(\tilde y{=}0\mid\mathbf{x})} = h\bigl(O(\mathbf{x})\bigr),
|
|
\qquad
|
|
h(O) = \frac{\rho + O(1-\rho)}{(1-\rho) + O\rho},
|
|
\]
|
|
and $h$ is strictly increasing for every $\rho < \tfrac12$.
|
|
|
|
Training with weight ratio $w$ on the noisy labels therefore does not threshold
|
|
$O(\mathbf{x})$ at $w$, it thresholds $h(O(\mathbf{x}))$ at $w$, which is equivalent to
|
|
thresholding the \emph{true} posterior odds at
|
|
\[
|
|
\eta_{\text{true}} = h^{-1}(w) = \frac{w(1-\rho) - \rho}{(1-\rho) - w\rho}.
|
|
\]
|
|
Whatever $w$ is chosen, the classifier ends up targeting $\eta_{\text{true}}$, not $w$,
|
|
and $\eta_{\text{true}}$ depends on $\rho$, which is generally unknown. Choosing $w$ no
|
|
longer means choosing $\eta$; it means choosing $\eta$ up to an unknown,
|
|
noise-dependent distortion.
|
|
|
|
\textsc{Roll}'s objective, and AUC-optimizing objectives more generally, sidestep
|
|
this mechanism entirely: they never form a weight ratio or compare a probability to a
|
|
fixed $\eta$. They target a metric directly from whichever labels they are given.
|
|
|
|
This relies specifically on the noise being class-conditional, $\rho$ a function of $y$
|
|
alone, independent of $\mathbf{x}$, as assumed throughout \Cref{sec:noisy-labels} and in
|
|
this thesis's noise experiments. The argument above only shows that this particular
|
|
distortion has this closed form under that assumption.
|
|
This does not show that likelihood-ratio ranking actually fails
|
|
under instance-dependent noise, only that the analysis given here does not, by itself,
|
|
extend to that setting.
|