diff --git a/content/method/method.tex b/content/method/method.tex new file mode 100644 index 0000000..0ea17ba --- /dev/null +++ b/content/method/method.tex @@ -0,0 +1,251 @@ + +\section{Problem Formulation} +\label{sec:problem-formulation} + +\subsection{Imbalanced Classification and the TPR/FPR Objective} +\label{sec:imbalanced-tpr-fpr} + +When fitting a classifier, it is most common to try and maximize the probability +of success. That is, given some dataset +$\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^{n}$ +with inputs $\mathbf{x}_i \in \mathcal{X}$ and binary labels $y_i \in \{0, 1\}$, we learn a scoring function $f_\theta : \mathcal{X} \to \mathbb{R}$ and apply a threshold $\tau$ to produce predictions $\hat{y} = \mathbf{1}[f_\theta(\mathbf{x}) > \tau]$. The standard objective is to minimize the empirical risk: + +$$\mathcal{L}(\theta) = \frac{1}{n} \sum_{i=1}^{n} \ell\!\left(f_\theta(\mathbf{x}_i),\, y_i\right)$$ + +for some surrogate loss $\ell$ (e.g. cross-entropy), which approximates maximizing classification accuracy $P(\hat{y} = y)$. + +For most applications, this optimization is warranted. However, there are many real-world settings in which +this type of fitting is ill-suited. + +One such application is in the case of inseparable classes. In many cases, no meaningful +separation boundry can be found between $\mathcal{D}_0$ and $\mathcal{D}_1$. In many cases, +especially in tabular datasets, there can be two data elements $i,j$ for which $x_i = x_j$, +however, $y_i \neq y_j$. Any deterministic classifier will have to decide whether to classify +inseperable points as one class or the other, but it remains hard to control the behavior of +a classifier while it is fitting to suggest one outcome or the other. + +The issue of inseparability becomes more apparent in certain applications where runtime is a key +and limiting factor. In many classification pipelines, a ``quick and dirty'' classifier may +filter out many candidates before an ``in depth'' classifier gives the final answer. This may be done +in events where running the full in-depth classification on all data points is not a cost-effective or time-efficient solution. +The ``quick and dirty'' classifier can be either low in parameters, making it run quickly. It may also only +make a quick decision based on part of the data point. For example, in the case of a classifier looking to flag +images of the ocean, a ``quick and dirty'' classifier may look at a very-low resolution version of the image and simply check if it is relatively ``blue''. It filters out many certain false candidates this way, then passes on the remainder to an in-depth, higher accuracy classifier. Solutions like this allow to run at near-optimal performance with lower compute resources in limited-time applications. These situations are common in large-volume image applications on the internet, as well as in astronomy/microscopy. + +Another such application is a case of highly imbalanced data. Even with separability, it is very +easy to a classifier to simply ``ignore'' the minority class. This may very likely be a local minima +while fitting, which may be inescapable. As was once said, ``a classifier that always returns 0 is +well fitted for flagging pictures of Michael Jordan bathing in green M&Ms''. Combined with inseparability, we see that in extreme cases, classifiers may choose to ignore, or otherwise penalize, the minority class. + +For every trained classifier, the ouput per model is a score (TODO rewrite and formalize). + +TODO finish this subsection + +- What usually happens + - Fitting + - graphing ROC and picking threshold based on tradeoff +- Usually we compensate for poor performance with + - Weighting between classes + - Possible other methods +- Problem with this method is lack of control. + + +% Motivate why standard loss functions (cross-entropy, MSE) break down under class +% imbalance: they optimize average accuracy, which a model can game by predicting +% the majority class. Introduce concrete examples (e.g. medical diagnosis, fraud +% detection) where the cost of a false negative vastly outweighs a false positive, +% and where the practitioner needs to operate at a specific FPR budget. Argue that +% what is actually needed is direct control over TPR at a fixed FPR, not a +% surrogate that only loosely correlates with it. + + + + +\subsection{Connection to Neyman-Pearson} +\label{sec:neyman-pearson} + +% Introduce the Neyman-Pearson lemma: among all tests at a given false-positive rate, +% the likelihood-ratio test maximizes TPR. Frame ROLL as a learnable, gradient-based +% realization of this principle — rather than assuming known class-conditional +% distributions (as classical NP does), ROLL estimates them from model scores during +% training. Briefly note that existing NP-inspired methods (covered in +% \Cref{chapter:related_work}) do not support deep learning / gradient-based +% optimization; this is the gap ROLL fills. + + +%------------------------------------------------ + +\section{The ROLL Framework} +\label{sec:roll-framework} + +In order to allow gradient-based learning methods to optimize TPR or FPR directly, we introduce +ROLL - Rate Optimized Likelyhood Loss. + +The core idea of ROLL is as follows: when training, the user selects that they would like to optimize +TPR given some fixed FPR. Then, during training, on every training step, the actual TPR is calculated for +the given FPR, and the fitting algorithm tries to maximize the TPR directly. This way, while training, we gain direct control over the performance at the threshold that we care about - avoiding needing to pick out a threshold later. + +In practice, however, things are not so simple. + +\subsection{The Differentiability Problem} +\label{sec:differentiability-problem} + +% TPR and FPR are defined via threshold comparisons on model scores, making them +% piecewise-constant functions of the model parameters with zero gradient almost +% everywhere — standard backpropagation cannot move through them. Show this +% concretely with the step-function definition of TPR/FPR and explain why +% common surrogates (AUC proxies, hinge-based losses) only indirectly target +% the operating point a practitioner cares about. + +Directly optimizing TPR for FPR, or vice versa, is an indefferentiable problem. + +For this sub-section, we will reffer to an example of maximizing TPR given a fixed FPR, +though this can be reversed. + +For any loss $\mathcal{L}(\theta)$, in order +to continue fitting our model, we must derive some gradient + +$\left{\partial \mathcal{L}(\theta)}{\partial m(x_i)}$ where $m(x_i)$ is the score +given to data point $x_i$ by our model $m$. +There are two inherent steps built-in to our methods: + +- Based on fixed rate, calculate threshold that gives said rate. +For example, when fixing FPR, find some threshold $\Theta$ s.t. TODO FORMALIZE +- Given the calculated threshold, calculate the rate we would like to maximize +( in our example, maximize TPR) + +Firstly, the TPR calculation is non-differentiable. To be more percise, the TPR as a function of $m(x_i)$ is +non-contiguous. If $m(x_i) \neq \Theta$, that is, the score is not exactly on the threshold, then changing it +by some $\varepsilon$ will not affect TPR at all, and the derivative of TPR w.r.t $m(x_i)$ will be zero. Likewise, +if $m(x_i) = \Theta$, the derivative of TPR w.r.t. $m(x_i)$ becomes inifinite. This derivative cannot be used +to continuously fit our model. + +Secondly, the derivative of the threshold, w.r.t the score, is also non-contiguous. TODO explain why. + + +\subsection{Score Distribution Fitting as a Solution} +\label{sec:score-distribution-fitting} + +% The key insight: instead of thresholding scores directly, fit a differentiable +% probability distribution to the positive-class and negative-class score outputs. +% TPR and FPR then become integrals of that distribution above/below a threshold, +% which are smooth and differentiable in the distribution parameters — and through +% them, in the model weights. Explain intuitively why this preserves the semantics +% of TPR/FPR while making the objective amenable to gradient descent. + +The way to overcome this problem in differentiability is not to calculate the TPR and FPR +directly, but to estimate them by fitting probability distributions to the score output of the model at +every step. Contiguous probability distributions are contiguous w.r.t their inputs. Additionally, we can calculate the TPR, FPR given the probability distributions CDF and ICDF functions. If we can gaurantee these functions to be differentiable, which proved to be the main challenge of this work, we can gaurantee this process to be differentiable. + + +\subsection{General ROLL Formulation and Derivation} +\label{sec:roll-formulation} + +% Present the abstract ROLL loss in terms of a generic distribution $\mathcal{F}$ +% fitted to model scores, without yet committing to a specific family. Derive the +% expression for TPR@FPR$_0$ as a function of $\mathcal{F}$'s parameters, and show +% that minimizing the loss is equivalent to maximizing TPR subject to the FPR +% constraint. Include the full derivation here; this is the core theoretical +% contribution of the chapter. + +Let us consider some dataset +$\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^{n}$ + +\subsection{Properties of the ROLL Formulation} +\label{sec:roll-properties} + +% State and prove (or sketch) key properties: consistency of the estimator as +% the number of samples grows, monotonicity of the loss with respect to the +% TPR/FPR trade-off, and behaviour at the constraint boundary. Discuss what +% guarantees ROLL provides and under what assumptions they hold. +% NOTE - these properties are auto-generated, consider suggestions. + + +\subsection{Gradient Computation and the Custom Backward Pass} +\label{sec:roll-backward} + +% Because the distribution parameters are themselves functions of model scores +% (not fixed), the backward pass must differentiate through the distribution +% fitting step. Describe the custom autograd implementation: how gradients flow +% from the ROLL loss back through the distribution parameters to the model +% weights. Highlight where standard autograd would fail and what the manual +% backward pass does differently. + + +%------------------------------------------------ + +\section{Distribution Instantiations} +\label{sec:roll-instantiations} + +% Each subsection below instantiates the general ROLL framework with a specific +% distributional assumption. Gaussian and Beta are included as parametric +% special cases; KDE is the primary contribution and receives the fullest treatment. + +\subsection{Gaussian ROLL} +\label{sec:roll-gaussian} + +% Assume positive- and negative-class scores are Gaussian distributed. The +% TPR/FPR integrals reduce to closed-form expressions in terms of the normal +% CDF, giving a simple and fast-to-compute instantiation of ROLL. Derive the +% loss and its gradient. Note the limitations: the Gaussian assumption can be +% violated in practice, motivating the more flexible instantiations below. + + +\subsection{Beta ROLL} +\label{sec:roll-beta} + +% When scores are passed through a sigmoid, they live in $[0,1]$ and a Beta +% distribution is a natural parametric fit. Derive the ROLL loss under the Beta +% assumption; the integrals involve the regularized incomplete beta function. +% Discuss when Beta ROLL is preferable to Gaussian ROLL and what its failure +% modes are. + + +\subsection{KDE ROLL} +\label{sec:roll-kde} + +% KDE background: see \Cref{sec:kde-background} in \Cref{chapter:related_work}. + +% The primary instantiation: use kernel density estimation to fit a non-parametric +% distribution to the score outputs, avoiding any parametric assumption about score +% shape. Derive the ROLL loss and gradients under KDE; the TPR/FPR integrals become +% weighted sums over kernel evaluations. Discuss the trade-off between flexibility +% and computational cost, and how the bandwidth (see \Cref{sec:kde-bandwidth}) +% controls the bias-variance trade-off of the estimator. + + +%------------------------------------------------ + +\section{Implementation Considerations} +\label{sec:roll-implementation} + +\subsection{Numerical Stability and the Scaling Trick} +\label{sec:roll-numerical-stability} + +% The KDE loss involves sums of kernel evaluations that can span many orders of +% magnitude, leading to floating-point underflow or overflow during training. +% Describe the scaling trick that normalizes these sums to a numerically stable +% range without changing the gradient direction, and show that it is equivalent +% to computing in log-space with a max-subtraction stabilizer (analogous to the +% log-sum-exp trick). + + +\subsection{Bandwidth Estimation for KDE} +\label{sec:kde-bandwidth} + +% The KDE bandwidth $h$ controls how smoothly the score distribution is estimated: +% too small and the loss becomes noisy; too large and the gradient signal is +% washed out. Describe the bandwidth estimation strategy used in ROLL (e.g. +% Silverman's rule, a learned or scheduled $h$), justify the choice, and discuss +% sensitivity of results to this hyperparameter. +% NOTE - not silverman's rule, we do use scheduling, but we also use different +% estimator - see impl. + + +% TODO: algorithm box — full training loop (forward → ROLL loss → custom backward → weight update) + + +%%% Local Variables: +%%% mode: latex +%%% TeX-master: "../../main" +%%% End: diff --git a/guidelines.md b/guidelines.md new file mode 100644 index 0000000..fb5477c --- /dev/null +++ b/guidelines.md @@ -0,0 +1,45 @@ +CS Thesis Structure (Rough Estimate) + +| # | Section | Pages | +|---|---------|-------| +| — | Abstract | 1 | +| 1 | **Introduction** — motivation, problem statement, contributions, thesis roadmap | 4–6 | +| 2 | **Background / Related Work** — prior methods, what you build on, what you differ from | 10–15 | +| 3 | **Method** — your actual contribution (ROLL loss, KDE formulation, gradient derivation) | 10–15 | +| 4 | **Experiments** — datasets, baselines, setup, results tables, figures | 12–18 | +| 5 | **Discussion** — what the results mean, failure modes, limitations | 4–6 | +| 6 | **Conclusion** — summary, future work | 2–3 | +| — | References | 3–6 | +| — | Appendix (optional) — proofs, extra tables, dataset details | 0–10 | + +**Total: ~50–80 pages** (varies a lot by field norms and advisor preference — ML/systems theses tend toward the lower end) + +--- + +A few notes for your project specifically: + +- **Method** will be the centerpiece — KDE formulation, the custom autograd backward pass, and how you frame the FPR-constrained objective deserve careful treatment. +- **Background** should cover: AUC/TPR@FPR optimization literature, KDE-based losses, imbalanced classification baselines (AUROC, focal loss, etc.). +- **Experiments** you already have a lot of infrastructure for — KEEL datasets + large datasets gives you a good breadth story. + +What's your advisor's expected page range, and do you have a target submission date? + +Claude> This is very helpful thank you. + +- Does page count include figures? +- Does page count include code sections? +- Are all proof in thesises in the appendix and only referenced inside the actual discussion section? + +Yes, yes, and it depends: + +**Figures** — yes, figures count toward page count. A results-heavy ML thesis can easily have 10+ pages of figures baked into those estimates above. + +**Code** — short snippets (pseudocode, a key algorithm block) go inline in the Method section and count toward pages. Full implementation code goes in the appendix or is omitted entirely — reviewers don't want to read your training loop. Pseudocode is almost always preferred over real code in the main body. + +**Proofs** — no fixed rule, but common practice: +- If the proof is **short and central** to understanding your contribution (e.g. showing your loss is differentiable), put it inline. +- If it's **long or technical** but important, put it in the appendix and write "proof in Appendix A" in the main text. +- If it's **a known result you're just citing**, skip the proof entirely and just cite the paper. + +For your thesis specifically: the KDE gradient derivation (the backward pass) is probably worth a compact inline treatment since it *is* the core technical contribution — but the full derivation details can go in an appendix. + diff --git a/main.tex b/main.tex index f154d5d..421ceaa 100644 --- a/main.tex +++ b/main.tex @@ -218,6 +218,11 @@ \addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{\chapnumforhebrewtoc}{סקירת ספרות}}{{\beginL\thepage\endL}}} \input{content/related_work/related_work} +\chapter{Method} +\label{chapter:method} +\addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{\chapnumforhebrewtoc}{שיטה}}{{\beginL\thepage\endL}}} +\input{content/method/method} + \chapter{Experimental Results} \label{chapter:experiments} \addtocontents{tocheb}{\contentsline{chapter}{\protect\numberline{\chapnumforhebrewtoc}{תוצאות ניסוייות}}{{\beginL\thepage\endL}}}