Citations, better flake.nix, better figure

bibliography/thesis.bib

@@ -1,3 +1,43 @@
+@article{tong2016neyman,
+  title={A survey on {Neyman--Pearson} classification and suggestions for future research},
+  author={Tong, Xin and Feng, Yang and Zhao, Anqi},
+  journal={{WIREs} Computational Statistics},
+  volume={8},
+  pages={64--81},
+  year={2016},
+  doi={10.1002/wics.1376},
+  publisher={Wiley}
+}
+
+@article{botev2010kde,
+  title={Kernel density estimation via diffusion},
+  author={Botev, Z. I. and Grotowski, J. F. and Kroese, D. P.},
+  journal={The Annals of Statistics},
+  volume={38},
+  number={5},
+  pages={2916--2957},
+  year={2010},
+  doi={10.1214/10-aos799},
+  publisher={Institute of Mathematical Statistics}
+}
+
+@book{press2007numerical,
+  title={Numerical Recipes: The Art of Scientific Computing},
+  author={Press, William H. and Teukolsky, Saul A. and Vetterling, William T. and Flannery, Brian P.},
+  edition={3rd},
+  year={2007},
+  publisher={Cambridge University Press},
+  address={Cambridge},
+  isbn={978-0-521-88068-8}
+}
+
+@article{chen2022beta,
+  title={Novel closed-form point estimators for the beta distribution},
+  author={Chen, Piao and Xiao, Xun},
+  journal={arXiv preprint arXiv:2210.05536},
+  year={2022}
+}
+
 @article{demaine2007jigsaw,
   title={Jigsaw puzzles, edge matching, and polyomino packing: Connections and complexity},
   author={Demaine, Erik D. and Demaine, Martin L.},

content/method/figures/roll_principle.py

@@ -92,12 +92,12 @@ ax2.axhline(F1_tau, color=C_POS, linestyle="-.", linewidth=0.9, alpha=0.7)
 ax2.plot(tau, F0_tau, "o", color=C_NEG, markersize=5, zorder=4)
 ax2.plot(tau, F1_tau, "o", color=C_POS, markersize=5, zorder=4)
 
-# Left-side y-axis labels for the intercepts
+# Right-side labels for the intercepts (placed where CDFs have flattened to ~1)
-x_min = x[0]
+x_label_right = x[-1] - 0.2
-ax2.text(x_min - 0.15, F0_tau, rf"$1-\alpha$",
+ax2.text(x_label_right, F0_tau, rf"$1-\alpha$",
-         fontsize=8, color=C_NEG, va="center", ha="right")
+         fontsize=8, color=C_NEG, va="bottom", ha="right")
-ax2.text(x_min - 0.15, F1_tau, r"$\hat{F}_1(\tau)$" + "\n(loss)",
+ax2.text(x_label_right, F1_tau, r"$\hat{F}_1(\tau)$" + "\n(loss)",
-         fontsize=7.5, color=C_POS, va="center", ha="right")
+         fontsize=7.5, color=C_POS, va="top", ha="right")
 
 # TPR annotation: double-headed arrow on the right + label
 x_bracket = tau + 1.9

content/method/method.tex

@@ -2,100 +2,49 @@
 \section{Problem Formulation}
 \label{sec:problem-formulation}
 
-\subsection{Imbalanced Classification and The TPR/FPR Objective}
+% Motivation for this section — why accuracy fails under class imbalance, why
-\label{sec:imbalanced-tpr-fpr}
+% TPR@FPR gives direct operating-point control, and the Neyman-Pearson framing
+% that underpins ROLL — is developed in \Cref{chapter:related_work}
+% (\Cref{sec:imbalanced-tpr-fpr,sec:neyman-pearson}).  This section establishes
+% only the notation and objective used throughout the chapter.
+Given a dataset $\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^{n}$ with inputs
+$\mathbf{x}_i \in \mathcal{X}$ and binary labels $y_i \in \{0, 1\}$, we learn a
+scoring function $f_\theta : \mathcal{X} \to \mathbb{R}$ by fitting.
+The positive and negative subsets of $\mathcal{D}$ are:
+\[
+  \mathcal{D}_1 = \{(\mathbf{x}_i, y_i) \in \mathcal{D} \mid y_i = 1\},
+  \qquad
+  \mathcal{D}_0 = \{(\mathbf{x}_i, y_i) \in \mathcal{D} \mid y_i = 0\}.
+\]
+Applying a threshold $\tau$ produces predictions $\hat{y} = \mathbf{1}[f_\theta(\mathbf{x}) > \tau]$.
+The \emph{true positive rate} (TPR) and \emph{false positive rate} (FPR) at $\tau$ are:
+\begin{align}
+  \operatorname{TPR}(\tau) &= P\!\bigl(f_\theta(\mathbf{x}) > \tau \mid y = 1\bigr), \\
+  \operatorname{FPR}(\tau) &= P\!\bigl(f_\theta(\mathbf{x}) > \tau \mid y = 0\bigr).
+\end{align}
+In our scenario, we either wish to learn $f_{\theta}$ in order to maximize $\operatorname{TPR}$
+subject to a fixed $\operatorname{FPR}$, or vice versa. Our fixed $\operatorname{TPR}$ or $\operatorname{FPR}$
+will be fixed to a value $\alpha \in (0, 1)$.
 
-% IMPROVEMENT SUGGESTIONS:
+Therefor, when maximizing our $\operatorname{TPR}$ subject to a fixed $\operatorname{FPR}$, we are solving for:
-% - Section title promises TPR/FPR but neither is formally defined here; the section
+\begin{equation}
-%   must land on a concrete statement of the TPR@FPR objective with proper notation.
+  \max_\theta \; \operatorname{TPR}\!\bigl(\tau^*(\theta)\bigr)
-% - The inseparability tangent (paragraphs 2-3) dilutes the argument; trim to one
+  \quad \text{s.t.} \quad \operatorname{FPR}\!\bigl(\tau^*(\theta)\bigr) = \alpha,
-%   sentence or move to Related Work. The cascade example in particular reads as
+  \label{eq:objective}
-%   background material unrelated to the TPR/FPR framing.
+\end{equation}
-% - Narrative arc is currently backwards: the comment block near the end (lines
-%   starting 'Motivate why standard loss functions...') is a better outline than
-%   the prose above it. Drive the section with: accuracy is gameable under imbalance
-%   -> class weighting is the standard fix -> it gives no direct control over the
-%   operating point -> therefore we need TPR@FPR directly.
-% - $\mathcal{D}_0$ and $\mathcal{D}_1$ are used but never defined; add definitions
-%   for the positive/negative subsets of $\mathcal{D}$ before first use.
-% - Suggested arc: (1) standard setup + accuracy objective; (2) why accuracy fails
-%   under imbalance -- the trivial majority-class classifier; (3) standard remedies
-%   and why they give only indirect control; (4) formally define TPR, FPR, ROC, and
-%   state the target objective: maximise TPR at a fixed FPR budget $\alpha$.
 
-When fitting a classifier, it is most common to try and maximize the probability
+where $\tau^*(\theta)$ is the threshold that achieves exactly $\operatorname{FPR} = \alpha$
-of success. That is, given some dataset
+for the current model $f_\theta$.
-$\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^{n}$
+Conversley, when we are minimizing $\operatorname{FPR}$ subject to a fixed $\operatorname{TPR}$ we are solving for:
-with inputs $\mathbf{x}_i \in \mathcal{X}$ and binary labels $y_i \in \{0, 1\}$, we learn a scoring function $f_\theta : \mathcal{X} \to \mathbb{R}$ and apply a threshold $\tau$ to produce predictions $\hat{y} = \mathbf{1}[f_\theta(\mathbf{x}) > \tau]$. The standard objective is to minimize the probability of classification error:
 
-$$\mathcal{L}(\theta) = \frac{1}{n} \sum_{i=1}^{n} \mathbf{1}[\hat{y}_i \neq y_i] = P(\hat{y} \neq y)$$
+\begin{equation}
-
+  \min_\theta \; \operatorname{FPR}\!\bigl(\tau^*(\theta)\bigr)
-which is equivalent to maximizing classification accuracy $P(\hat{y} = y)$.
+  \quad \text{s.t.} \quad \operatorname{TPR}\!\bigl(\tau^*(\theta)\bigr) = \alpha,
-
+  \label{eq:objective}
-For most applications, this optimization is warranted. However, there are many real-world settings in which
+\end{equation}
-this type of fitting is ill-suited.
-
-One such application is in the case of inseparable classes. In many cases, no meaningful
-separation boundary can be found between $\mathcal{D}_0$ and $\mathcal{D}_1$. In many cases,
-especially in tabular datasets, there can be two data elements $i,j$ for which $\mathbf{x}_i = \mathbf{x}_j$,
-however, $y_i \neq y_j$. Any deterministic classifier will have to decide whether to classify
-inseparable points as one class or the other, but it remains hard to control the behavior of
-a classifier while it is fitting to suggest one outcome or the other.
-
-The issue of inseparability becomes more apparent in certain applications where runtime is a key
-and limiting factor. In many classification pipelines, a ``quick and dirty'' classifier may
-filter out many candidates before an ``in depth'' classifier gives the final answer. This may be done
-in events where running the full in-depth classification on all data points is not a cost-effective or time-efficient solution.
-The ``quick and dirty'' classifier can be either low in parameters, making it run quickly. It may also only
-make a quick decision based on part of the data point. For example, in the case of a classifier looking to flag
-images of the ocean, a ``quick and dirty'' classifier may look at a very-low resolution version of the image and simply check if it is relatively ``blue''. It filters out many certain false candidates this way, then passes on the remainder to an in-depth, higher accuracy classifier. Solutions like this allow to run at near-optimal performance with lower compute resources in limited-time applications. These situations are common in large-volume image applications on the internet, as well as in astronomy/microscopy.
-
-Another such application is a case of highly imbalanced data. Even with separability, it is very
-easy to a classifier to simply ``ignore'' the minority class. This may very likely be a local minimum
-while fitting, which may be inescapable. As was once said, ``a classifier that always returns 0 is
-well fitted for flagging pictures of Michael Jordan bathing in green M\&Ms''. Combined with inseparability, we see that in extreme cases, classifiers may choose to ignore, or otherwise penalize, the minority class.
-
-For every trained classifier, the output per model is a score (TODO rewrite and formalize).
-
-TODO finish this subsection
-
-\begin{itemize}
-    \item What usually happens
-    \begin{itemize}
-        \item Fitting
-        \item Graphing ROC and picking threshold based on tradeoff
-    \end{itemize}
-    \item Usually we compensate for poor performance with
-    \begin{itemize}
-        \item Weighting between classes
-        \item Possible other methods
-    \end{itemize}
-    \item Problem with this method is lack of control.
-\end{itemize}
-
-
-% Motivate why standard loss functions (cross-entropy, MSE) break down under class
-% imbalance: they optimize average accuracy, which a model can game by predicting
-% the majority class. Introduce concrete examples (e.g. medical diagnosis, fraud
-% detection) where the cost of a false negative vastly outweighs a false positive,
-% and where the practitioner needs to operate at a specific FPR budget. Argue that
-% what is actually needed is direct control over TPR at a fixed FPR, not a
-% surrogate that only loosely correlates with it.
-
-
-
-
-\subsection{Connection to Neyman-Pearson}
-\label{sec:neyman-pearson}
-
-% Introduce the Neyman-Pearson lemma: among all tests at a given false-positive rate,
-% the likelihood-ratio test maximizes TPR. Frame ROLL as a learnable, gradient-based
-% realization of this principle — rather than assuming known class-conditional
-% distributions (as classical NP does), ROLL estimates them from model scores during
-% training. Briefly note that existing NP-inspired methods (covered in
-% \Cref{chapter:related_work}) do not support deep learning / gradient-based
-% optimization; this is the gap ROLL fills.
 
+The ROLL framework, introduced in
+\Cref{sec:roll-framework}, provides a method of differentiable surrogate for these objectives.
 
 %------------------------------------------------
 
@@ -109,7 +58,7 @@ The core idea of ROLL is as follows: when training, the user selects that they w
 TPR given some fixed FPR. Then, during training, on every training step, the actual TPR is calculated for
 the given FPR, and the fitting algorithm tries to maximize the TPR directly. This way, while training, we gain direct control over the performance at the threshold that we care about - avoiding needing to pick out a threshold later.
 
-In Neyman-Pearson terms, our algorithm wishes to directly minimize the Type-1 error given a fixed Type-2 error, or vice versa.
+In Neyman-Pearson terms~\cite{tong2016neyman}, our algorithm wishes to directly minimize the Type-1 error given a fixed Type-2 error, or vice versa.
 
 In practice, however, things are not so simple.
 
@@ -482,23 +431,72 @@ and letting $\tau = \hat{F}_0^{-1}(\alpha) = \mu_0 + \sigma_0\sqrt{2}\,\operator
 \subsection{Beta ROLL}
 \label{sec:roll-beta}
 
-% When scores are passed through a sigmoid they live in $[0,1]$ and a Beta
+Beta ROLL applies when model scores are constrained to the open interval $(0,1)$,
-% distribution is a natural parametric fit. The integrals involve the
+as is the case when $f_\theta$ uses a sigmoid output activation.  The Beta distribution
-% regularised incomplete beta function. Discuss when Beta ROLL is preferable
+is supported on $(0,1)$ and its two shape parameters allow it to capture a wide range
-% to Gaussian ROLL and what its failure modes are.
+of unimodal and skewed score distributions that a Gaussian cannot represent.
 
 \subsubsection{Forward}
 \label{sec:roll-beta-forward}
 
-% Instantiate \eqref{eq:roll-tpr-at-fpr} under Beta-distributed scores.
+Given score vectors $\mathbf{s}^{(k)} \in (0,1)^{|\mathcal{B}_k|}$ for class $k$, let
-% Parameters $a_k, b_k > 0$ are fitted by MLE (or method of moments).
+$\mathbf{t}^{(k)} = \mathbf{1} - \mathbf{s}^{(k)}$ denote the complementary scores.
-% The CDF is the regularised incomplete beta function $I_x(a, b)$.
+For any vector $\mathbf{v}$, write $\bar{v} = \frac{1}{|\mathbf{v}|}\sum_i v_i$.
+We estimate the shape parameters using the closed-form log-moment estimator
+of~\cite{chen2022beta}: define for each class $k$,
+
+\begin{equation}
+  D_k = \overline{s^{(k)}\!\ln s^{(k)}} - \bar{s}^{(k)}\cdot\overline{\ln s^{(k)}}
+      + \overline{t^{(k)}\!\ln t^{(k)}} - \bar{t}^{(k)}\cdot\overline{\ln t^{(k)}}
+  \label{eq:beta-denom}
+\end{equation}
+
+\begin{equation}
+  \hat{a}_k = \frac{\bar{s}^{(k)}}{D_k}, \qquad \hat{b}_k = \frac{\bar{t}^{(k)}}{D_k}
+  \label{eq:beta-params}
+\end{equation}
+
+These estimators follow from two simultaneous moment conditions:
+$\mathbb{E}[X] = a/(a+b)$ and
+$\operatorname{Cov}(X,\ln X) + \operatorname{Cov}(Y,\ln Y) = 1/(a+b)$,
+where $Y=1-X\sim\operatorname{Beta}(b,a)$.
+Replacing expectations by sample means and solving in closed form yields~\eqref{eq:beta-params}.
+
+The CDF of $\operatorname{Beta}(\hat{a}_k,\hat{b}_k)$ at $x\in(0,1)$ is the
+regularised incomplete beta function:
+\begin{equation}
+  \hat{F}_k(x) = I_x(\hat{a}_k,\hat{b}_k)
+               = \frac{B(x;\,\hat{a}_k,\hat{b}_k)}{B(\hat{a}_k,\hat{b}_k)},
+  \qquad B(x;\,a,b) = \int_0^x u^{a-1}(1-u)^{b-1}\,\mathrm{d}u
+  \label{eq:beta-cdf}
+\end{equation}
+
+Substituting into the general ROLL framework \eqref{eq:roll-tpr-at-fpr}:
+\begin{equation}
+  \mathcal{L}_{\text{ROLL-TPR@FPR}}^{\text{BETA}}(f_\theta(\mathcal{B});\,\alpha)
+  = I_{\hat\tau}(\hat{a}_1,\hat{b}_1),
+  \qquad
+  \hat\tau = I^{-1}_\alpha(\hat{a}_0,\hat{b}_0)
+  \label{eq:roll-tpr-at-fpr-beta}
+\end{equation}
+where $I^{-1}_\alpha(a,b)$ denotes the $\alpha$-quantile of $\operatorname{Beta}(a,b)$.
 
 \subsubsection{Gradient Derivation}
 \label{sec:roll-beta-backward}
 
-% Differentiate through $I_x$ and its inverse. Both have known derivatives;
+Unlike the Gaussian case, we do not derive closed-form gradients for Beta ROLL.
-% note any numerical considerations when $a$ or $b$ are near zero.
+The regularised incomplete beta function $I_x(a,b)$ and its inverse are smooth
+functions of all their arguments and are available as differentiable operations in
+PyTorch (\texttt{torch.special.betainc} and \texttt{torch.special.betaincinv}).
+Gradients of \eqref{eq:roll-tpr-at-fpr-beta} with respect to $f_\theta(\mathbf{x}_i)$
+are therefore propagated via automatic differentiation through the estimator
+\eqref{eq:beta-params}, the CDF evaluation, and the quantile inversion.
+
+Note that the gradient balance property (\Cref{prop:gradient-balance}) does not hold
+for Beta ROLL: the sigmoid activation that maps raw scores into $(0,1)$ is nonlinear,
+so a uniform additive shift of the raw scores produces a non-uniform shift of the
+transformed inputs, violating the translation-invariance assumption required by the
+proof.
 
 
 \subsection{KDE ROLL}
@@ -545,20 +543,20 @@ The CDF of the KDE function then becomes:
 \]
 
 While the CDF has a nice formula, the inverse CDF has no known closed formula. It instead must be calculated numerically. Thus, the calculation of the decision threshold $\tau$ must be achevied
-using a numerical calculation algorithm. In this work, Newton Raphson is chosen.
+using a numerical calculation algorithm. In this work, the Newton--Raphson method~\cite{press2007numerical} is chosen.
 
 Firstly we must calculate $\hat{F}_0^{-1}(\alpha)$ = \tau. We start with our initial guess, $\tau_0$, and for each step, calculate:
 
 \[
-  \tau_{n+1} = \tau_n - \frac{\hat{F}_0(\tau_n ; \mathcal{X}_0)}{\frac{\partial \hat{F}_0(\tau_n ; \mathcal{X}_0)}{\partial \tau}}
+  \tau_{n+1} = \tau_n - \frac{\hat{F}_0(\tau_n ; \mathcal{X}_0) - \alpha}{\frac{\partial \hat{F}_0(\tau_n ; \mathcal{X}_0)}{\partial \tau}}
 \]
 
-This step is taken until $\hat{F}_0(\tau_n ; \mathcal{B}_0) \approx \alpha$ within some acceptable error (in our case $1e-3$).
+This step is taken until $\hat{F}_0(\tau_n ; \mathcal{B}_0) \approx \alpha$ within some acceptable error (in our case $1e-4$).
 
 Our derivative w.r.t. $\tau$ is
 
 \[                              %
-  \frac{\partial \hat{F}_0(\tau_n ; \mathcal{X}_0)}{\partial \tau} = \frac{1}{\mathcal{X}_0}\sum_i \sigma'(\tau - x_i)
+  \frac{\partial \hat{F}_0(\tau_n ; \mathcal{X}_0)}{\partial \tau} = \frac{1}{|\mathcal{X}_0|}\sum_i \sigma'(\tau - x_i)
 \]
 
 At this point, we have succesfully calculated $\tau$ for which $\tau = \hat{F}_0^{-1}(\alpha ; \mathbf{X}_0)$.
@@ -586,15 +584,18 @@ and has no effect on $\tau$ (which depends only on $\mathcal{B}_0$). Therefore:
 
 \[
   \dfrac{\partial \hat{F}_1(\hat{F}_0^{-1}(\alpha))}{\partial f_{\theta}(\mathbf{x}_i)} =
-  \frac{\partial \frac{1}{|\mathcal{B}_1|}\sum_{\mathbf{x}_j \in \mathcal{B}_1} \sigma_1(f_{\theta}(\mathbf{x}_j) - \tau)}{\partial f_{\theta}(\mathbf{x}_i)}
+  \frac{\partial \frac{1}{|\mathcal{B}_1|}\sum_{\mathbf{x}_j \in \mathcal{B}_1} \sigma_1(\tau - f_{\theta}(\mathbf{x}_j))}{\partial f_{\theta}(\mathbf{x}_i)}
 \]
 
 \[
-  = \frac{\partial \left(\sum_{\mathbf{x}_j \in \mathcal{B}_1 ; j \neq i} \frac{1}{|\mathcal{B}_1|}\sigma_1(f_{\theta}(\mathbf{x}_j) - \tau) + \frac{1}{|\mathcal{B}_1|}\sigma_1(f_{\theta}(\mathbf{x}_i) - \tau)\right)}{\partial f_{\theta}(\mathbf{x}_i)}
+  = \frac{\partial \left(\sum_{\mathbf{x}_j \in \mathcal{B}_1 ; j \neq i} \frac{1}{|\mathcal{B}_1|}\sigma_1(\tau - f_{\theta}(\mathbf{x}_j)) + \frac{1}{|\mathcal{B}_1|}\sigma_1(\tau - f_{\theta}(\mathbf{x}_i))\right)}{\partial f_{\theta}(\mathbf{x}_i)}
 \]
 
+Applying the chain rule to the only $f_\theta(\mathbf{x}_i)$-dependent term,
+$\frac{\partial}{\partial f_\theta(\mathbf{x}_i)}(\tau - f_\theta(\mathbf{x}_i)) = -1$:
+
 \begin{equation}
-  \frac{\partial \hat{F}_1(\hat{F}_0^{-1}(\alpha))}{\partial f_{\theta}(\mathbf{x}_i)} = \frac{1}{|\mathcal{B}_1|}\sigma_1'(f_{\theta}(\mathbf{x}_i) - \tau)
+  \frac{\partial \hat{F}_1(\hat{F}_0^{-1}(\alpha))}{\partial f_{\theta}(\mathbf{x}_i)} = -\frac{1}{|\mathcal{B}_1|}\sigma_1'(\tau - f_{\theta}(\mathbf{x}_i))
   \label{eq:kde-grad-y1}
 \end{equation}
 
@@ -607,11 +608,13 @@ each factor of the chain rule separately.
 \medskip\noindent\textit{Derivative of $\hat{F}_1$ with respect to $\tau$.}
 
 \[
-  \frac{\partial \hat{F}_1(\hat{F}_0^{-1}(\alpha))}{\partial \hat{F}_0^{-1}(\alpha)} = \frac{\partial \frac{1}{|\mathcal{B}_1|}\sum_{\mathbf{x}_j \in \mathcal{B}_1} \sigma_1(f_{\theta}(\mathbf{x}_j) - \tau)}{\partial \tau}
+  \frac{\partial \hat{F}_1(\hat{F}_0^{-1}(\alpha))}{\partial \hat{F}_0^{-1}(\alpha)} = \frac{\partial \frac{1}{|\mathcal{B}_1|}\sum_{\mathbf{x}_j \in \mathcal{B}_1} \sigma_1(\tau - f_{\theta}(\mathbf{x}_j))}{\partial \tau}
 \]
 
+Applying the chain rule, $\frac{\partial}{\partial\tau}(\tau - f_\theta(\mathbf{x}_j)) = +1$:
+
 \begin{equation}
-  = -\frac{1}{|\mathcal{B}_1|}\sum_{\mathbf{x}_j \in \mathcal{B}_1}\sigma_1'(f_{\theta}(\mathbf{x}_j) - \tau)
+  = +\frac{1}{|\mathcal{B}_1|}\sum_{\mathbf{x}_j \in \mathcal{B}_1}\sigma_1'(\tau - f_{\theta}(\mathbf{x}_j))
   \label{eq:kde-dF1-dtau}
 \end{equation}
 
@@ -670,6 +673,41 @@ by $\sigma_0'(\tau - f_{\theta}(\mathbf{x}_i))$:
   \label{eq:kde-grad-tau}
 \end{equation}
 
+\medskip\noindent\textit{Efficient computation via the sigmoid identity.}
+
+Direct evaluation of \eqref{eq:kde-grad-tau} requires computing $\sigma_0'$ for every
+point, then dividing by the per-point value — potentially unstable when a point is far
+from $\tau$. For the sigmoid kernel, we can avoid this by exploiting the identity
+
+\begin{equation}
+  \sigma(u;\,v)\,\bigl(1 - \sigma(u;\,v)\bigr) = \frac{\sigma'(u;\,v)}{v}
+  \label{eq:sigmoid-identity}
+\end{equation}
+
+which holds for all $u \in \mathbb{R}$ (proved by direct substitution: both sides equal
+$\exp(-v|u|)/(1+\exp(-v|u|))^2$). Applying this with $u = \tau - f_\theta(\mathbf{x}_j)$:
+
+\[
+  \sigma_0'(\tau - f_\theta(\mathbf{x}_j)) = v_0\,\sigma_0(\tau - f_\theta(\mathbf{x}_j))
+  \,\bigl(1-\sigma_0(\tau - f_\theta(\mathbf{x}_j))\bigr)
+\]
+
+Since $v_0$ cancels between numerator and denominator of \eqref{eq:kde-grad-tau},
+the ratio can be written entirely in terms of the sigmoid values already computed
+during the forward pass:
+
+\begin{equation}
+  \frac{\partial \hat{F}_0^{-1}(\alpha ; \mathcal{B}_0)}{\partial f_\theta(\mathbf{x}_i)}
+  =
+  \frac{\sigma_0(\tau-f_\theta(\mathbf{x}_i))\,\bigl(1-\sigma_0(\tau-f_\theta(\mathbf{x}_i))\bigr)}
+       {\displaystyle\sum_{\mathbf{x}_j \in \mathcal{B}_0}
+         \sigma_0(\tau-f_\theta(\mathbf{x}_j))\,\bigl(1-\sigma_0(\tau-f_\theta(\mathbf{x}_j))\bigr)}
+  \label{eq:kde-grad-tau-efficient}
+\end{equation}
+
+This avoids recomputing $\sigma_0'$ (which requires an extra exponential), instead
+reusing the sigmoid activations cached from the CDF computation.
+
 \paragraph{Combined gradient.}
 
 Substituting \Cref{eq:kde-grad-y1}, \Cref{eq:kde-dF1-dtau}, and \Cref{eq:kde-grad-tau}
@@ -678,9 +716,9 @@ into \Cref{eq:roll-gradient}:
 \begin{equation}
   \frac{\partial \mathcal{L}_{\text{ROLL-TPR@FPR}}}{\partial f_\theta(\mathbf{x}_i)} =
   \begin{cases}
-    \dfrac{1}{|\mathcal{B}_1|}\,\sigma_1'(f_{\theta}(\mathbf{x}_i) - \tau)
+    -\dfrac{1}{|\mathcal{B}_1|}\,\sigma_1'(\tau - f_{\theta}(\mathbf{x}_i))
       & \text{if } y_i = 1 \\[14pt]
-    -\dfrac{1}{|\mathcal{B}_1|}\!\displaystyle\sum_{\mathbf{x}_j \in \mathcal{B}_1}\!\sigma_1'(f_{\theta}(\mathbf{x}_j) - \tau)
+    +\dfrac{1}{|\mathcal{B}_1|}\!\displaystyle\sum_{\mathbf{x}_j \in \mathcal{B}_1}\!\sigma_1'(\tau - f_{\theta}(\mathbf{x}_j))
     \;\cdot\;
     \dfrac{\sigma_0'(\tau - f_{\theta}(\mathbf{x}_i))}{\displaystyle\sum_{\mathbf{x}_j \in \mathcal{B}_0}\sigma_0'(\tau - f_{\theta}(\mathbf{x}_j))}
       & \text{if } y_i = 0
@@ -690,6 +728,88 @@ into \Cref{eq:roll-gradient}:
 
 where $\tau = \hat{F}_0^{-1}(\alpha ; \mathcal{B}_0)$.
 
+\subsubsection{Sigmoid Kernel: Explicit Equations}
+\label{sec:roll-kde-sigmoid}
+
+The derivations above hold for any differentiable CDF kernel $\sigma$.
+In our implementation we choose the logistic sigmoid:
+%
+\begin{equation}
+  \sigma(x ; v) = \frac{1}{1 + \exp(-vx)},
+  \qquad
+  \sigma'(x ; v) = \frac{v\,\exp(-v|x|)}{\bigl(1 + \exp(-v|x|)\bigr)^{2}}
+  \label{eq:sigmoid-kernel}
+\end{equation}
+%
+where $v > 0$ is the bandwidth parameter.  The derivative formula evaluates at
+$|x|$ rather than $x$: since $\sigma'$ is an even function
+($\sigma'(x) = \sigma'(-x)$ for any symmetric logistic), both forms are
+mathematically identical, but $\exp(-v|x|) \to 0$ as $|x| \to \infty$ while
+$\exp(-vx) \to \infty$ for $x \to -\infty$, so the $|x|$ form avoids
+floating-point overflow when scores fall well below the threshold.
+
+\paragraph{Forward pass.}
+
+Substituting \eqref{eq:sigmoid-kernel} into the KDE CDF:
+%
+\begin{equation}
+  \hat{F}_{k}(\tau ; \mathbf{s}^{(k)}) =
+  \frac{1}{|\mathcal{B}_k|}
+  \sum_{\mathbf{x}_j \in \mathcal{B}_k}
+  \frac{1}{1 + \exp\!\bigl(-v_k\bigl(\tau - f_\theta(\mathbf{x}_j)\bigr)\bigr)}
+  \label{eq:kde-sigmoid-cdf}
+\end{equation}
+%
+and its derivative with respect to $\tau$ — used both in the Newton--Raphson
+inversion and in the gradient computation below — is:
+%
+\begin{equation}
+  \frac{\partial \hat{F}_{k}(\tau)}{\partial \tau}
+  =
+  \frac{1}{|\mathcal{B}_k|}
+  \sum_{\mathbf{x}_j \in \mathcal{B}_k}
+  \frac{v_k\,\exp\!\bigl(-v_k\bigl|\tau - f_\theta(\mathbf{x}_j)\bigr|\bigr)}
+       {\Bigl(1 + \exp\!\bigl(-v_k\bigl|\tau - f_\theta(\mathbf{x}_j)\bigr|\bigr)\Bigr)^{2}}
+  \label{eq:kde-sigmoid-pdf}
+\end{equation}
+
+\paragraph{Gradient.}
+
+Define the sigmoid kernel PDF shorthand:
+\begin{equation}
+  \varphi_k(u) \;=\;
+  \frac{v_k\,\exp(-v_k|u|)}{\bigl(1+\exp(-v_k|u|)\bigr)^{2}}
+  \label{eq:sigmoid-phi}
+\end{equation}
+This is simply $\sigma'(u ; v_k)$ from \eqref{eq:sigmoid-kernel}, a bell-shaped function
+centred at zero with width controlled by bandwidth $v_k$.
+Substituting \eqref{eq:sigmoid-kernel} into \eqref{eq:kde-grad-combined} and using this
+notation, the explicit sigmoid-kernel gradient is:
+
+\begin{equation}
+  \frac{\partial \mathcal{L}_{\text{ROLL-TPR@FPR}}}{\partial f_\theta(\mathbf{x}_i)} =
+  \begin{cases}
+    -\dfrac{\varphi_1\!\left(\tau - f_\theta(\mathbf{x}_i)\right)}{|\mathcal{B}_1|}
+    & \text{if } y_i = 1\\[14pt]
+    +\,\dfrac{\displaystyle\sum_{\mathbf{x}_j \in \mathcal{B}_1}
+                \varphi_1\!\left(\tau - f_\theta(\mathbf{x}_j)\right)}{|\mathcal{B}_1|}
+    \;\cdot\;
+    \dfrac{\varphi_0\!\left(\tau-f_\theta(\mathbf{x}_i)\right)}
+          {\displaystyle\sum_{\mathbf{x}_j \in \mathcal{B}_0}
+             \varphi_0\!\left(\tau-f_\theta(\mathbf{x}_j)\right)}
+    & \text{if } y_i = 0
+  \end{cases}
+  \label{eq:kde-sigmoid-grad}
+\end{equation}
+
+where $\tau = \hat{F}_0^{-1}(\alpha ; \mathcal{B}_0)$.
+The signs reflect the direction of each class's effect on the loss: raising a
+positive-class score pushes $\hat{F}_1(\tau)$ down (gradient negative), while
+raising a negative-class score shifts $\tau$ right, also lowering $\hat{F}_1(\tau)$
+(gradient positive). Positive-class samples are weighted by their individual kernel
+PDF at $\tau$; negative-class samples are weighted by the mean positive-class kernel
+PDF scaled by each sample's share of the negative-class kernel mass.
+
 %------------------------------------------------
 
 \subsection{Properties of the ROLL Formulation}
@@ -879,12 +999,12 @@ concentration of gradient mass around $\tau$, with smooth decay on either side.
 When computing KDE, kernel selection and bandwidth estimation affect the final result greatly. Large bandwidths
 don't accurately track the probability distribution, while small kernels result in a very jagged CDF, which for us means very low gradient values unless unless a point is very close to the decision threshold.
 
-Use of the Improved Sheather Jones method of kernel density estimation worked reliably well at creating very close-tracking, yet smooth CDFs.
+Use of the Improved Sheather-Jones (ISJ) method~\cite{botev2010kde} of bandwidth selection worked reliably well at creating very close-tracking, yet smooth CDFs.
 
 \subsubsection{Bandwidth Scaling for KDE}
 \label{sec:kde-bandwidth-scaling}
 
-While the Improved Sheather Jones method worked very well for selecting an acceptable bandwidth, this wasn't necessarily wanted during training. Near the start of the training session, scores tended to be all low. This naturally made the selected bandwidth to be too low. This caused very low training performance near the start of a training session, which could result in non-convergence of the model even after significant training epochs elapsed.
+While the ISJ method worked very well for selecting an acceptable bandwidth, this wasn't necessarily wanted during training. Near the start of the training session, scores tended to be all low. This naturally made the selected bandwidth to be too low. This caused very low training performance near the start of a training session, which could result in non-convergence of the model even after significant training epochs elapsed.
 
 The solution was to use a bandwidth scheduler. Starting with large bandwidths resulted in gradients that spread out more evenly across all scores, and less non-convergence runs occurred. The bandwidth scheduler artificially scales up the bandwidth at the beginning to speed up initial training, and then tapers back to restore the intended bandwidth in order to maximize resuslts.
 

content/related_work/related_work.tex

@@ -1,3 +1,112 @@
+% ============================================================
+% PLACEMENT: near the START of Ch. 2, before KDE and AUC/lit
+% review sections.  This is the reader's first encounter with
+% the problem setting and the TPR@FPR objective.  The arc
+% should be: (1) standard accuracy under imbalance fails;
+% (2) class-weighting is the usual fix but gives no direct
+% operating-point control; (3) define TPR, FPR, ROC formally;
+% (4) state the target: maximise TPR at a fixed FPR budget α.
+% ============================================================
+\section{Imbalanced Classification and The TPR/FPR Objective}
+\label{sec:imbalanced-tpr-fpr}
+
+% IMPROVEMENT SUGGESTIONS (carried over from original draft):
+% - Narrative arc is currently backwards.  Drive the section
+%   with: accuracy is gameable under imbalance -> class
+%   weighting is the standard fix -> it gives no direct
+%   control over the operating point -> therefore we need
+%   TPR@FPR directly.
+% - The inseparability tangent (cascade-classifier example)
+%   dilutes the argument; trim to one sentence or cut.
+% - $\mathcal{D}_0$ and $\mathcal{D}_1$ must be defined
+%   before first use.
+% - Land the section on a concrete, formal statement of the
+%   TPR@FPR objective with proper notation.
+
+When fitting a classifier, it is most common to try and maximize the probability
+of success. That is, given some dataset
+$\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^{n}$
+with inputs $\mathbf{x}_i \in \mathcal{X}$ and binary labels $y_i \in \{0, 1\}$, we learn a scoring function $f_\theta : \mathcal{X} \to \mathbb{R}$ and apply a threshold $\tau$ to produce predictions $\hat{y} = \mathbf{1}[f_\theta(\mathbf{x}) > \tau]$. The standard objective is to minimize the probability of classification error:
+
+$$\mathcal{L}(\theta) = \frac{1}{n} \sum_{i=1}^{n} \mathbf{1}[\hat{y}_i \neq y_i] = P(\hat{y} \neq y)$$
+
+which is equivalent to maximizing classification accuracy $P(\hat{y} = y)$.
+
+For most applications, this optimization is warranted. However, there are many real-world settings in which
+this type of fitting is ill-suited.
+
+One such application is in the case of inseparable classes. In many cases, no meaningful
+separation boundary can be found between $\mathcal{D}_0$ and $\mathcal{D}_1$. In many cases,
+especially in tabular datasets, there can be two data elements $i,j$ for which $\mathbf{x}_i = \mathbf{x}_j$,
+however, $y_i \neq y_j$. Any deterministic classifier will have to decide whether to classify
+inseparable points as one class or the other, but it remains hard to control the behavior of
+a classifier while it is fitting to suggest one outcome or the other.
+
+The issue of inseparability becomes more apparent in certain applications where runtime is a key
+and limiting factor. In many classification pipelines, a ``quick and dirty'' classifier may
+filter out many candidates before an ``in depth'' classifier gives the final answer. This may be done
+in events where running the full in-depth classification on all data points is not a cost-effective or time-efficient solution.
+The ``quick and dirty'' classifier can be either low in parameters, making it run quickly. It may also only
+make a quick decision based on part of the data point. For example, in the case of a classifier looking to flag
+images of the ocean, a ``quick and dirty'' classifier may look at a very-low resolution version of the image and simply check if it is relatively ``blue''. It filters out many certain false candidates this way, then passes on the remainder to an in-depth, higher accuracy classifier. Solutions like this allow to run at near-optimal performance with lower compute resources in limited-time applications. These situations are common in large-volume image applications on the internet, as well as in astronomy/microscopy.
+
+Another such application is a case of highly imbalanced data. Even with separability, it is very
+easy to a classifier to simply ``ignore'' the minority class. This may very likely be a local minimum
+while fitting, which may be inescapable. As was once said, ``a classifier that always returns 0 is
+well fitted for flagging pictures of Michael Jordan bathing in green M\&Ms''. Combined with inseparability, we see that in extreme cases, classifiers may choose to ignore, or otherwise penalize, the minority class.
+
+For every trained classifier, the output per model is a score (TODO rewrite and formalize).
+
+TODO finish this section
+
+\begin{itemize}
+    \item What usually happens
+    \begin{itemize}
+        \item Fitting
+        \item Graphing ROC and picking threshold based on tradeoff
+    \end{itemize}
+    \item Usually we compensate for poor performance with
+    \begin{itemize}
+        \item Weighting between classes
+        \item Possible other methods
+    \end{itemize}
+    \item Problem with this method is lack of control.
+\end{itemize}
+
+
+% Motivate why standard loss functions (cross-entropy, MSE) break down under class
+% imbalance: they optimize average accuracy, which a model can game by predicting
+% the majority class. Introduce concrete examples (e.g. medical diagnosis, fraud
+% detection) where the cost of a false negative vastly outweighs a false positive,
+% and where the practitioner needs to operate at a specific FPR budget. Argue that
+% what is actually needed is direct control over TPR at a fixed FPR, not a
+% surrogate that only loosely correlates with it.
+
+
+% ============================================================
+% PLACEMENT: near the END of Ch. 2, after KDE and AUC/lit
+% review sections, just before the chapter summary.  This
+% section bridges from prior work to the method chapter by
+% framing ROLL as a learnable, gradient-based NP test.
+% ============================================================
+\section{Connection to Neyman-Pearson}
+\label{sec:neyman-pearson}
+
+% Introduce the Neyman-Pearson lemma: among all tests at a
+% given false-positive rate, the likelihood-ratio test
+% maximizes TPR.  Frame ROLL as a learnable, gradient-based
+% realization of this principle — rather than assuming known
+% class-conditional distributions (as classical NP does),
+% ROLL estimates them from model scores during training.
+% Briefly note that existing NP-inspired methods do not
+% support deep learning / gradient-based optimization; this
+% is the gap ROLL fills.
+
+TODO write this section.
+
+
+%------------------------------------------------
+
 \section{topic a}
 \label{sec:related_work:jigsaw_puzzles}
 \addcontentsline{tocheb}{section}{\protect\numberline{\secnumforhebrewtoc}{נושא א}}

flake.nix

@@ -62,6 +62,7 @@
           (pkgs.texlive.combine texPkgs)
           pkgs.culmus
           pkgs.fontconfig
+          pkgs.poppler-utils
           (pkgs.python3.withPackages (ps: [ ps.sympy ps.numpy ps.matplotlib ps.scipy ]))
         ];
         shellHook = ''