228 lines
11 KiB
TeX
228 lines
11 KiB
TeX
% ============================================================
|
||
% PLACEMENT: near the START of Ch. 2, before KDE and AUC/lit
|
||
% review sections. This is the reader's first encounter with
|
||
% the problem setting and the TPR@FPR objective. The arc
|
||
% should be: (1) standard accuracy under imbalance fails;
|
||
% (2) class-weighting is the usual fix but gives no direct
|
||
% operating-point control; (3) define TPR, FPR, ROC formally;
|
||
% (4) state the target: maximise TPR at a fixed FPR budget α.
|
||
% ============================================================
|
||
\section{Imbalanced Classification and The TPR/FPR Objective}
|
||
\label{sec:imbalanced-tpr-fpr}
|
||
|
||
% IMPROVEMENT SUGGESTIONS (carried over from original draft):
|
||
% - Narrative arc is currently backwards. Drive the section
|
||
% with: accuracy is gameable under imbalance -> class
|
||
% weighting is the standard fix -> it gives no direct
|
||
% control over the operating point -> therefore we need
|
||
% TPR@FPR directly.
|
||
% - The inseparability tangent (cascade-classifier example)
|
||
% dilutes the argument; trim to one sentence or cut.
|
||
% - $\mathcal{D}_0$ and $\mathcal{D}_1$ must be defined
|
||
% before first use.
|
||
% - Land the section on a concrete, formal statement of the
|
||
% TPR@FPR objective with proper notation.
|
||
|
||
When fitting a classifier, it is most common to try and maximize the probability
|
||
of success. That is, given some dataset
|
||
$\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^{n}$
|
||
with inputs $\mathbf{x}_i \in \mathcal{X}$ and binary labels $y_i \in \{0, 1\}$, we learn a scoring function $f_\theta : \mathcal{X} \to \mathbb{R}$ and apply a threshold $\tau$ to produce predictions $\hat{y} = \mathbf{1}[f_\theta(\mathbf{x}) > \tau]$. The standard objective is to minimize the probability of classification error:
|
||
|
||
$$\mathcal{L}(\theta) = \frac{1}{n} \sum_{i=1}^{n} \mathbf{1}[\hat{y}_i \neq y_i] = P(\hat{y} \neq y)$$
|
||
|
||
which is equivalent to maximizing classification accuracy $P(\hat{y} = y)$.
|
||
|
||
For most applications, this optimization is warranted. However, there are many real-world settings in which
|
||
this type of fitting is ill-suited.
|
||
|
||
One such application is in the case of inseparable classes. In many cases, no meaningful
|
||
separation boundary can be found between $\mathcal{D}_0$ and $\mathcal{D}_1$. In many cases,
|
||
especially in tabular datasets, there can be two data elements $i,j$ for which $\mathbf{x}_i = \mathbf{x}_j$,
|
||
however, $y_i \neq y_j$. Any deterministic classifier will have to decide whether to classify
|
||
inseparable points as one class or the other, but it remains hard to control the behavior of
|
||
a classifier while it is fitting to suggest one outcome or the other.
|
||
|
||
The issue of inseparability becomes more apparent in certain applications where runtime is a key
|
||
and limiting factor. In many classification pipelines, a ``quick and dirty'' classifier may
|
||
filter out many candidates before an ``in depth'' classifier gives the final answer. This may be done
|
||
in events where running the full in-depth classification on all data points is not a cost-effective or time-efficient solution.
|
||
The ``quick and dirty'' classifier can be either low in parameters, making it run quickly. It may also only
|
||
make a quick decision based on part of the data point. For example, in the case of a classifier looking to flag
|
||
images of the ocean, a ``quick and dirty'' classifier may look at a very-low resolution version of the image and simply check if it is relatively ``blue''. It filters out many certain false candidates this way, then passes on the remainder to an in-depth, higher accuracy classifier. Solutions like this allow to run at near-optimal performance with lower compute resources in limited-time applications. These situations are common in large-volume image applications on the internet, as well as in astronomy/microscopy.
|
||
|
||
Another such application is a case of highly imbalanced data. Even with separability, it is very
|
||
easy to a classifier to simply ``ignore'' the minority class. This may very likely be a local minimum
|
||
while fitting, which may be inescapable. As was once said, ``a classifier that always returns 0 is
|
||
well fitted for flagging pictures of Michael Jordan bathing in green M\&Ms''. Combined with inseparability, we see that in extreme cases, classifiers may choose to ignore, or otherwise penalize, the minority class.
|
||
|
||
For every trained classifier, the output per model is a score (TODO rewrite and formalize).
|
||
|
||
TODO finish this section
|
||
|
||
\begin{itemize}
|
||
\item What usually happens
|
||
\begin{itemize}
|
||
\item Fitting
|
||
\item Graphing ROC and picking threshold based on tradeoff
|
||
\end{itemize}
|
||
\item Usually we compensate for poor performance with
|
||
\begin{itemize}
|
||
\item Weighting between classes
|
||
\item Possible other methods
|
||
\end{itemize}
|
||
\item Problem with this method is lack of control.
|
||
\end{itemize}
|
||
|
||
|
||
% Motivate why standard loss functions (cross-entropy, MSE) break down under class
|
||
% imbalance: they optimize average accuracy, which a model can game by predicting
|
||
% the majority class. Introduce concrete examples (e.g. medical diagnosis, fraud
|
||
% detection) where the cost of a false negative vastly outweighs a false positive,
|
||
% and where the practitioner needs to operate at a specific FPR budget. Argue that
|
||
% what is actually needed is direct control over TPR at a fixed FPR, not a
|
||
% surrogate that only loosely correlates with it.
|
||
|
||
|
||
% ============================================================
|
||
% PLACEMENT: near the END of Ch. 2, after KDE and AUC/lit
|
||
% review sections, just before the chapter summary. This
|
||
% section bridges from prior work to the method chapter by
|
||
% framing ROLL as a learnable, gradient-based NP test.
|
||
% ============================================================
|
||
\section{Connection to Neyman-Pearson}
|
||
\label{sec:neyman-pearson}
|
||
|
||
% Introduce the Neyman-Pearson lemma: among all tests at a
|
||
% given false-positive rate, the likelihood-ratio test
|
||
% maximizes TPR. Frame ROLL as a learnable, gradient-based
|
||
% realization of this principle — rather than assuming known
|
||
% class-conditional distributions (as classical NP does),
|
||
% ROLL estimates them from model scores during training.
|
||
% Briefly note that existing NP-inspired methods do not
|
||
% support deep learning / gradient-based optimization; this
|
||
% is the gap ROLL fills.
|
||
|
||
TODO write this section.
|
||
|
||
|
||
%------------------------------------------------
|
||
|
||
\section{topic a}
|
||
\label{sec:related_work:jigsaw_puzzles}
|
||
\addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{נושא א}}
|
||
|
||
To be continued.
|
||
|
||
|
||
\section{topic b}
|
||
\label{sec:related_work:relaxation_labeling}
|
||
\addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{נושא ב}}
|
||
|
||
To be continued.
|
||
|
||
\subsection{sub topic b.1}
|
||
\label{subsec:formulation_as_rl:rationale:type_2}
|
||
\addcontentsline{tocheb}{subsection}{\protect\numberline{\subsecnumforhebrewtoc}{תת נושא ב1}}
|
||
|
||
To be continued.
|
||
|
||
\begin{figure}[H]
|
||
\centering
|
||
|
||
\begin{subfigure}[b]{0.3\textwidth}
|
||
\begin{tikzpicture}
|
||
\node[anchor=south west, inner sep=0] at (0,0) {\includegraphics[width=\textwidth]{content/related_work/images/2x2_puzzle_grid.png}};
|
||
\draw[step=0.5\textwidth] (0,0) grid (\textwidth,\textwidth);
|
||
\node[font=\large] at (0.25\textwidth,0.75\textwidth) {(1,1)};
|
||
\node[font=\large] at (0.75\textwidth,0.75\textwidth) {(1,2)};
|
||
\node[font=\large] at (0.25\textwidth,0.25\textwidth) {(2,1)};
|
||
\node[font=\large] at (0.75\textwidth,0.25\textwidth) {(2,2)};
|
||
\end{tikzpicture}
|
||
\caption{}
|
||
\label{fig:type_1_goal_and_labeling:dimensions}
|
||
\end{subfigure}
|
||
\hfill
|
||
\begin{subfigure}[b]{0.3\textwidth}
|
||
\begin{tikzpicture}
|
||
\node[anchor=south west, inner sep=0] at (0,0) {\includegraphics[width=\textwidth]{content/related_work/images/2x2puzzle_type_1.png}};
|
||
\draw[step=0.5\textwidth] (0,0) grid (\textwidth,\textwidth);
|
||
\node[font=\large, color=red] at (0.25\textwidth,0.75\textwidth) {Piece 1};
|
||
\node[font=\large, color=red] at (0.75\textwidth,0.75\textwidth) {Piece 2};
|
||
\node[font=\large, color=red] at (0.25\textwidth,0.25\textwidth) {Piece 3};
|
||
\node[font=\large, color=red] at (0.75\textwidth,0.25\textwidth) {Piece 4};
|
||
\end{tikzpicture}
|
||
\caption{}
|
||
\label{fig:type_1_goal_and_labeling:pieces}
|
||
\end{subfigure}
|
||
\hfill
|
||
\begin{subfigure}[b]{0.3\textwidth}
|
||
\begin{tikzpicture}
|
||
\node[anchor=south west, inner sep=0] at (0,0) {\includegraphics[width=\textwidth]{content/related_work/images/2x2puzzle_solution.png}};
|
||
\draw[step=0.5\textwidth] (0,0) grid (\textwidth,\textwidth);
|
||
|
||
\node[font=\large, color=red] at (0.25\textwidth,0.75\textwidth) {Piece 3};
|
||
|
||
\node[font=\large, color=red] at (0.75\textwidth,0.75\textwidth) {Piece 2};
|
||
|
||
\node[font=\large, color=red] at (0.25\textwidth,0.25\textwidth) {Piece 4};
|
||
|
||
\node[font=\large, color=red] at (0.75\textwidth,0.25\textwidth) {Piece 1};
|
||
\end{tikzpicture}
|
||
\caption{}
|
||
\label{fig:type_1_goal_and_labeling:solution}
|
||
\end{subfigure}
|
||
\vfill
|
||
\begin{subfigure}[b]{1\textwidth}
|
||
\centering
|
||
|
||
\begin{tikzpicture}
|
||
\def\scaletitles{0.88}
|
||
\def\minimumEntrySize{0.95cm}
|
||
|
||
\matrix[matrix of nodes,
|
||
nodes={draw, align=center, minimum size=\minimumEntrySize},
|
||
row 1/.style={nodes={draw=none, gray, font=\footnotesize, scale=\scaletitles}},
|
||
column 1/.style={nodes={draw=none, gray, font=\footnotesize,
|
||
scale=\scaletitles}}]
|
||
{
|
||
\node{}; &
|
||
\node{(1,1)}; &
|
||
\node{(1,2)}; &
|
||
\node{(2,1)}; &
|
||
\node{(2,2)};\\
|
||
%----------%
|
||
\node{Piece 1}; &
|
||
\node{0}; & \node{0}; & \node{0}; & \node[text=blue]{1};\\
|
||
%----------%
|
||
\node{Piece 2}; &
|
||
\node{0}; & \node[text=blue]{1}; & \node{0}; & \node{0};\\
|
||
%----------%
|
||
\node{Piece 3}; &
|
||
\node[text=blue]{1}; & \node{0}; & \node{0}; & \node{0};\\
|
||
%----------%
|
||
\node{Piece 4}; &
|
||
\node{0}; & \node{0}; & \node[text=blue]{1}; & \node{0};\\
|
||
};
|
||
\end{tikzpicture}
|
||
\caption{}
|
||
\label{fig:type_1_goal_and_labeling:labeling}
|
||
\end{subfigure}
|
||
\caption[fig A - Example for fig]{Some example}
|
||
\label{fig:type_1_goal_and_labeling}
|
||
\end{figure}
|
||
|
||
|
||
\begin{table}[H]
|
||
\centering
|
||
\begin{tabular}{ |c|c|c|c|c|c| }
|
||
\hline
|
||
\multicolumn{1}{|c|}{Puzzle Type} & Direct & Neighbor & Perfect & Occupied & Feasible \\
|
||
\hline
|
||
Type 1 & {0\%} & {0.1\%} & 0 & {1\%} & 0 \\
|
||
\hline
|
||
Type 2 & {0.001\%} & {0.1\%} & 0 & {1.2\%} & 0 \\
|
||
\hline
|
||
\end{tabular}
|
||
\caption[Table A - Example for table]{Some example}
|
||
\label{table:plain_rl_results}
|
||
\end{table} |