backup: 2026-10-05 09:48
This commit is contained in:
+375
-4
@@ -28,6 +28,9 @@ This distinction must be stated clearly in Ch. 3 and Ch. 2 related work.
|
||||
- sec:imbalanced-tpr-fpr and sec:neyman-pearson were removed from this chapter and live in Ch. 2
|
||||
- Both objectives (TPR@FPR and FPR@TPR) are equivalent by negating scores and swapping labels;
|
||||
all derivations given for TPR@FPR only — FPR@TPR follows by transformation
|
||||
- EXCEPTION to the above (2026-09-10): sec:np-class-weighting and sec:np-noise-invariance are
|
||||
NP-theoretic content that DOES live in this chapter now, despite NP itself being background —
|
||||
see the dedicated section below for why and where.
|
||||
|
||||
** Section Structure
|
||||
|
||||
@@ -38,14 +41,358 @@ This distinction must be stated clearly in Ch. 3 and Ch. 2 related work.
|
||||
| Differentiability Problem | sec:differentiability-problem | Written |
|
||||
| Score Distribution Fitting | sec:score-distribution-fitting | Written |
|
||||
| General ROLL Formulation and Derivation | sec:roll-formulation | Written — core theoretical contribution |
|
||||
| Gaussian ROLL (forward + gradient) | sec:roll-gaussian | Written |
|
||||
| Beta ROLL (forward + gradient) | sec:roll-beta | Stub — TBD |
|
||||
| Distribution Instantiations (intro only) | sec:roll-instantiations | Rewritten — KDE only, parametric → appendix |
|
||||
| KDE ROLL (forward + gradient) | sec:roll-kde | Written |
|
||||
| Properties | sec:roll-properties | Written — gradient balance + locality |
|
||||
| Properties | sec:roll-properties | Written — gradient balance + locality + sign correctness |
|
||||
| Custom Backward Pass | sec:roll-backward | Stub — TBD |
|
||||
| Numerical Stability | sec:roll-numerical-stability | Stub — TBD |
|
||||
| Numerical Stability | sec:roll-numerical-stability | Written — normalization invariance proposition + frozen-c convention (2026-09-09) |
|
||||
| Bandwidth Estimation (ISJ + scheduling) | sec:kde-bandwidth | Written |
|
||||
|
||||
sec:np-class-weighting and sec:np-noise-invariance are NOT in this chapter as of the final
|
||||
2026-09-10 move -- they live in Appendix L now (appendix:np-theory), pointed to from
|
||||
Discussion 5.1.1. See the dedicated section below for the full story; don't re-add them
|
||||
here if seen in stale context.
|
||||
|
||||
Gaussian and Beta subsections removed from Ch. 3 body; full math lives in appendix:
|
||||
- Gaussian (forward + analytic gradient): appendix:gauss-grad (appendix_gauss_grad.tex, expanded)
|
||||
- Beta (forward + autograd note): appendix:beta-roll (appendix_beta_roll.tex, new)
|
||||
|
||||
** Appendix Structure (after 2026-08-29 restructure, +2 appendices added 2026-09-09)
|
||||
|
||||
| Label | Chapter title | File | Hebrew letter |
|
||||
|---|---|---|---|
|
||||
| appendix:gauss-grad | Gaussian ROLL: Forward Pass and Gradient Derivation | appendix_gauss_grad.tex | נספח ב |
|
||||
| appendix:beta-roll | Beta ROLL: Forward Pass and Gradient | appendix_beta_roll.tex | נספח ג |
|
||||
| appendix:kde-grad | KDE ROLL: Gradient Derivation | appendix_kde_grad.tex | נספח ד (now also has sec:kde-grad-stability, 2026-09-09) |
|
||||
| appendix:kde-sigmoid | KDE ROLL: Sigmoid Kernel — Explicit Equations | appendix_kde_sigmoid.tex | נספח ה |
|
||||
| appendix:kde-bisection | KDE ROLL: Bisection for CDF Inversion | appendix_kde_bisection.tex | נספח ו |
|
||||
| appendix:kde-nr | KDE ROLL: Newton–Raphson Inversion (Superseded) | appendix_kde_nr.tex | נספח ז |
|
||||
| appendix:roll-code | ROLL Loss: Implementation Listing | appendix_roll_code.tex | נספח ח |
|
||||
| appendix:keel-noise | KEEL Benchmark: Noise Tables at 5% and 10% | appendix_keel_noise.tex | נספח ח (verify) |
|
||||
| appendix:grad-balance | Proof of the Gradient Balance Proposition | appendix_grad_balance.tex | נספח ט |
|
||||
| appendix:bandwidth-freeze | KDE ROLL: Adaptive-Bandwidth Gradients and the Freezing Convention | appendix_bandwidth_freeze.tex | נספח י (appended at end, 2026-09-09) |
|
||||
| appendix:score-normalization | Score Normalisation: Invariance, ISJ, and the Frozen Chain Rule | appendix_score_normalization.tex | נספח יא (appended at end, 2026-09-09) |
|
||||
| appendix:np-theory | Theoretical Justification for AUC-Optimizing Losses via the Neyman-Pearson Lemma | appendix_np_theory.tex | נספח יב (appended at end, 2026-09-10; NOT ROLL-derived math, holds sec:np-class-weighting/sec:np-noise-invariance, pointed to from Discussion 5.1.1) |
|
||||
|
||||
Note: not every audit fix gets its own appendix — small, self-contained formula-level
|
||||
fixes (like the sigmoid-derivative stability fix) go directly into the existing relevant
|
||||
appendix as a new paragraph/section; only fixes needing real counterexamples/propositions
|
||||
(bandwidth freezing, score normalization) got whole new appendix chapters. Keep it that
|
||||
way — the user pushed back once already on the thesis getting too long.
|
||||
|
||||
** Mentor Audit Progress (2026-09-07 audit, thesis_mathematical_audit.pdf)
|
||||
|
||||
Full narrative for each item (derivations, counterexamples, code changes, test results)
|
||||
lives in [[id:151159e8-1105-4a84-af9c-72d31c5e0660][thesis/todos]] under "Mentor mathematical audit fixes" — not duplicated here.
|
||||
This audit list is now FULLY CLOSED (all items DONE as of 2026-09-09).
|
||||
|
||||
DONE (2026-09-09), in priority order: bandwidth backprop convention (frozen v_0,v_1 +
|
||||
appendix:bandwidth-freeze) → bisection stopping criterion (certified-loss bound,
|
||||
appendix:kde-bisection + roll.py fix) → bisection bracket range (exact target-dependent
|
||||
bracket, same files) → score-normalization contradiction (normalization-invariance
|
||||
proposition + frozen-c convention, appendix:score-normalization) → cached-sigmoid
|
||||
derivative underflow (log-space/softmax fix, sec:kde-grad-stability in appendix_kde_grad.tex
|
||||
+ roll.py `_log_sigma_prime`) → indicator-derivative-at-jump wording (undefined, not
|
||||
infinite, + finite-sample-vs-population-rate distinction) → score/label reversal scope
|
||||
(footnote: negation trick needs reflection-equivariance; Beta uses s→1−s instead, verified
|
||||
exact via the â↔b̂ swap identity) → softened "structural property" overclaim in Ch. 5's
|
||||
noise-robustness discussion.
|
||||
|
||||
CORRECTED framing of the common thread (2026-09-09 — user caught my oversimplification,
|
||||
do NOT put the old version in the thesis): it is NOT "freeze anything numerically
|
||||
estimated" — tau is also numerically estimated (via bisection) and we deliberately DO
|
||||
differentiate through it, since that's the entire mechanism by which negative-class
|
||||
scores get any gradient at all. The real distinction: differentiate through quantities
|
||||
that are part of the TRUE objective's own definition (tau — TPR@FPR is defined via "the
|
||||
threshold achieving this FPR," so tau's score-dependence is structural, not an artifact);
|
||||
freeze quantities that exist ONLY because of the smoothing/approximation scheme used to
|
||||
make the true (indicator-based) objective differentiable in the first place, and have no
|
||||
counterpart in that true objective at all (v_0, v_1 bandwidth; c normalization scale).
|
||||
User's framing: bandwidth is "an estimation of our approximation of a theoretical
|
||||
distribution... could be wrong and flawed, and produces artifacts" when differentiated
|
||||
through — that's the right way to say it. This principle is NOT yet written into the
|
||||
thesis (user wants it kept out of the discussion chapter for now, maybe later) — this
|
||||
memory entry is for continuity across sessions only.
|
||||
|
||||
** Third Mentor Review (2026-09-10, screenshots) — AOC divisor mismatch
|
||||
|
||||
Separate from the "second mentor review" 5-point list below (different reviewer/round,
|
||||
delivered as screenshots so no direct file/page cite -- mentor referred to it loosely as
|
||||
"equation 3.13" but the actual affected equation turned out to be eq:roll-aoc, now (3.15)).
|
||||
|
||||
Mentor's claim: thesis defines L_ROLL-AOC with normalization 1/99 (eq:roll-aoc,
|
||||
method.tex), but the implementation's _AOC_DIVISOR is 100. Mentor's argument: grid
|
||||
spacing is 0.01 (i.e. 1/100), so a Riemann-sum approximation should divide by 100,
|
||||
assuming the omitted alpha=1 endpoint is zero.
|
||||
|
||||
VERIFIED: real mismatch, confirmed directly in impl/src/roll.py (sibling repo, NOT under
|
||||
thesis/ -- lives at /home/aner/projects/research-projects/roll/impl/src/roll.py):
|
||||
`_AOC_DIVISOR = 100` (line 114), used in BOTH the forward pass (`total_loss =
|
||||
losses.sum() / divisor`, line 296) and the backward-pass gradients (lines 320, 333) --
|
||||
code's own comment even states "Sum divided by 100 approximates integral [sec:roll-aoc]",
|
||||
i.e. the code was ALWAYS intended to match the thesis and the thesis just had the wrong
|
||||
constant written down. Since the code is what actually produced every reported result,
|
||||
the FIX belongs in the thesis text, not the code.
|
||||
|
||||
IMPORTANT NUANCE (checked numerically before agreeing with the mentor's full framing):
|
||||
g(1) = 1-TPR(FPR^-1(1)) = 1-TPR(-infinity) = 0 EXACTLY (verified: thresholding at -infinity
|
||||
classifies everything positive) -- so (1/100)*sum_{i=1}^{99} IS a complete, valid
|
||||
right-Riemann sum over 100 equal subintervals of [0,1] (the "missing" 100th term is known
|
||||
to be exactly zero, not actually omitted). This gives a clean, PROVABLE error bound of
|
||||
exactly 1/100=0.01 (classical monotone-function Riemann-sum bound: error <= range*step =
|
||||
(g(0)-g(1))*(1/100) <= 1*(1/100)), tighter and cleaner than the old "1/99≈0.0101" claim.
|
||||
BUT: numerically stress-tested both /99 and /100 against the true integral on 200 random
|
||||
Gaussian-type AOC curves, and /99 had SMALLER actual error than /100 in ALL 200 trials
|
||||
(though both stayed well under the 0.01 worst-case bound, ~0.005 typical). So it is NOT
|
||||
mathematically true that "100 is more accurate than 99" for realistic AOC integrands --
|
||||
it's simply that 100 is what the code computes and is a cleanly-justifiable, provably-
|
||||
bounded choice, not a superior one. Told the user this nuance directly before fixing,
|
||||
rather than just conceding "99 was a bug, 100 is better" at face value.
|
||||
|
||||
FIX applied (method.tex, sec:roll-aoc), all three affected spots:
|
||||
1. eq:roll-aoc (now eq 3.15): 1/99 -> 1/100.
|
||||
2. Darboux-bound paragraph right after it: rewritten around the g(1)=0-exact fact and the
|
||||
right-Riemann-sum framing, replacing the old approximate "1/99≈0.0101 by a standard
|
||||
Darboux-sum argument" with the new EXACT derivation and clean 0.01 bound (shown as
|
||||
|L_ROLL-AOC - L_AOC| <= (g(0)-g(1))*(1/100) <= 1/100 = 0.01).
|
||||
3. eq:roll-aoc-gradient (now eq 3.16): 1/99 -> 1/100 (gradient of a linear combination
|
||||
scales the same way as the combination itself).
|
||||
Checked for other stale "1/99" or "99-point average" mentions elsewhere in the thesis
|
||||
(appendix_gauss_grad.tex, discussion_and_conclusions.tex, method.tex intro) -- all other
|
||||
occurrences just say "99-point" or "99 evenly-spaced" descriptively (correctly describing
|
||||
the NUMBER of evaluated points, unaffected by the divisor value), none needed changes.
|
||||
Rebuilt thesis clean (nix build exit 0), confirmed eq 3.15/3.16 both show 1/100 and the
|
||||
new bound derivation renders correctly in the compiled PDF.
|
||||
|
||||
** Second Mentor Review (2026-09-09) — 5 points, status
|
||||
|
||||
New round of feedback, distinct from the first audit. ALL 5 POINTS NOW RESOLVED as of
|
||||
2026-09-10 (episode-count justification, the last open one, closed same day as this note).
|
||||
Full detail in [[id:151159e8-1105-4a84-af9c-72d31c5e0660][thesis/todos]]; summary here:
|
||||
1. Justify varying episode counts — RESOLVED 2026-09-10. User's corrections/actions:
|
||||
CIFAR-10N was actually already n=5 (a stale "n=3" caption, not a real 3-episode run) —
|
||||
just a caption fix, no data change. ChestXray14 and Food-101N WERE genuinely at n=3 —
|
||||
user ran 2 more real episodes each and updated the mean/std numbers in
|
||||
tab:results:large accordingly (git diff showed real numeric changes to those two rows
|
||||
only, e.g. ChestXray14 ROLL-AOC 0.839->0.838, Food-101N 0.867->0.867 with updated std
|
||||
devs) before asking me to verify entry + rewrite the counts. I caught one formatting
|
||||
nit while verifying: Food-101N/Focal cell had "$\pm$0.01" (2 decimals) vs 3 decimals
|
||||
everywhere else in that table -- fixed to "$\pm$0.010".
|
||||
Final state: n=5 EVERYWHERE except KEEL noise experiments (5%/10%/20%), which stay at
|
||||
n=10, with justification added inline in sec:experiments:configurations (experiments.tex
|
||||
~line 76): "KEEL's datasets are small enough that each episode trains quickly despite
|
||||
the number of datasets involved, making it practical to run more episodes there for a
|
||||
more reliable noise-robustness comparison" -- user's own justification, given verbatim
|
||||
in the request ("keel dataset benchmarks ran very fast, despite their number, so they
|
||||
ran for more episodes"). Also fixed the two stale table captions (tab:results:cifar10n
|
||||
"n=3"->"n=5"; tab:results:large dropped the "Food-101N and ChestXray14: n=3" exception
|
||||
entirely, now just states "n=5 episodes" for the whole table). Rebuilt thesis clean (nix
|
||||
build exit 0), confirmed every episode-count mention in the compiled PDF is now n=5
|
||||
except the three KEEL-noise tables (4.3, H.1, H.2), which correctly stay n=10.
|
||||
2. AUC integral definition: same variable (alpha) on page 5 (Ch.2) vs Ch.3? — RESOLVED
|
||||
2026-09-10. Was a real inconsistency: TPR is defined everywhere else (eq 3.1, eq 3.6) as
|
||||
a function of a THRESHOLD tau; FPR^-1(alpha) is the separate function mapping an FPR
|
||||
level to a threshold; Ch.2's page-5 definition and eq:objective-auc (eq 3.5) correctly
|
||||
write the composition TPR(FPR^-1(alpha)), but sec:roll-aoc (method.tex, starting ~line
|
||||
300) silently dropped the FPR^-1 wrapper and wrote bare "TPR(alpha)" six separate times
|
||||
(lines 300, 303, 306, 324, 325, and the fig:roll-aoc-roc caption) -- overloading TPR to
|
||||
take an FPR-level directly, contradicting its own prior definition. Not a computational
|
||||
bug (the AUC value computed is identical either way) but a real notation-consistency
|
||||
flaw -- exactly the kind of thing a careful reader/mentor would catch. Fix: global
|
||||
replace_all of the literal string "\operatorname{TPR}(\alpha)" ->
|
||||
"\operatorname{TPR}(\operatorname{FPR}^{-1}(\alpha))" across method.tex (safe as a
|
||||
blanket replace since grep confirmed all 6 occurrences in the whole file needed the
|
||||
same fix, and no correctly-already-composed instance used that exact bare substring).
|
||||
Note: $\mathcal{L}_{\text{ROLL-TPR@FPR}}(\alpha)$ notation right next to these was NOT
|
||||
part of the problem -- that loss was already, by its own original definition
|
||||
(eq:roll-tpr-at-fpr), a function of alpha directly, so no inconsistency there; only the
|
||||
TPR symbol specifically was being overloaded. Rebuilt thesis clean (nix build exit 0),
|
||||
confirmed all instances now render as "TPR(FPR⁻¹(α))" consistently throughout the PDF
|
||||
(eq 3.5, eq 3.13, sec:roll-aoc prose, and the figure caption all match).
|
||||
3. Theoretical justification for noise robustness — RESOLVED 2026-09-09, see below.
|
||||
4. Exact gradient or stop-gradient approximation for the backward pass? — ALREADY FIXED,
|
||||
pre-existing text. method.tex:470-476 explicitly: "We adopt this as a deliberate
|
||||
stop-gradient convention"; appendix:bandwidth-freeze derives the fully-adaptive (exact)
|
||||
gradient alongside specifically to show what's traded away. No ambiguity.
|
||||
5. "Chapters A and B" should say "Appendix" — WAS NOT FIXED, now IS. Root cause: this
|
||||
wasn't literal hardcoded text (source already used \Cref{appendix:...}) — it was a
|
||||
cleveref config bug. Appendices are typeset via \chapter, so cleveref classified every
|
||||
appendix label under the generic "chapter" counter and printed "Chapter X" everywhere
|
||||
(~10+ instances across the whole PDF: "Chapter A", "Chapters A and B", "Chapter I",
|
||||
"Chapter H", etc.), even though main.tex:170-171 already correctly defined
|
||||
\Crefname{appendix}{Appendix}{Appendices} — that declaration was just never triggered.
|
||||
Fix: added \crefalias{chapter}{appendix} right after \appendix in main.tex. Rebuilt and
|
||||
confirmed every appendix ref now says "Appendix", the 5 real chapters still say
|
||||
"Chapter N" (crefalias only affects labels defined after it in source order, and no
|
||||
appendix file refers back to a main chapter{:...} label, so this was safe).
|
||||
|
||||
** Neyman-Pearson Theory: now Appendix L, pointed to from Discussion 5.1.1
|
||||
|
||||
Answers second-mentor point 3 ("is there a theoretical justification for noise
|
||||
robustness?"). Has moved TWICE now (2026-09-10, same day, both moves user-initiated):
|
||||
Ch.2 (related_work.tex, original) -> Ch.3 (method.tex, inside sec:roll-formulation) ->
|
||||
final location: Appendix L (content/appendixs/appendix_np_theory.tex, new chapter added
|
||||
2026-09-10, label appendix:np-theory, \crefalias makes it render as "Appendix L" not
|
||||
"Chapter L" per the cleveref fix). Do not be surprised by the two intermediate locations if
|
||||
they show up in old context/diffs -- Appendix L + Discussion 5.1.1 is the FINAL, current
|
||||
state; Ch.2 and Ch.3 no longer contain this material at all.
|
||||
|
||||
User's reasoning for the final move: read the Ch.3-embedded version and felt it was still
|
||||
in the wrong place -- genuinely new theoretical content shouldn't be buried inline in the
|
||||
Method chapter either. Final structure: (1) Appendix L holds the full two-section analysis
|
||||
verbatim (sec:np-class-weighting = L.1, sec:np-noise-invariance = L.2, both \section now,
|
||||
not \subsection, matching appendix-chapter convention where each appendix's top-level
|
||||
divisions are \section); (2) Discussion sec:discussion:np-justification (new subsection,
|
||||
FIRST subsection of Section 5.1 "Discussion", before "Noise Robustness of ROLL-AOC") is a
|
||||
short pointer paragraph, wording lifted close to verbatim from what the user dictated,
|
||||
ending "The full analysis is given in \Cref{appendix:np-theory}"; (3) Ch.3 (method.tex) has
|
||||
ZERO remaining trace -- fully reverted to how it read before any of this was added; (4) Ch.2
|
||||
sec:neyman-pearson also has zero trace, ends at "The practical gap" as it always did.
|
||||
|
||||
Generalization the user explicitly asked to be stated: this is NOT ROLL-specific. It
|
||||
justifies any order-dependent/ranking-based loss -- primarily ROLL-AOC, but also LibAUC and
|
||||
similar AUC-optimizing methods -- since none of them ever form a class-weight ratio or
|
||||
compare to a fixed eta in the first place. Added this framing to: the appendix's own lead-in
|
||||
paragraph, the end of L.1 (explicit LibAUC mention), and the Discussion pointer paragraph.
|
||||
|
||||
Appendix L.1's "ROLL avoids this" sentence now says "The TPR@FPR objective of
|
||||
\Cref{sec:roll-formulation}, \eqref{eq:roll-tpr-at-fpr}" (explicit chapter-crossing
|
||||
reference, since the appendix is no longer inside the section that defines it) -- still
|
||||
resolves correctly cross-chapter via \eqref/\Cref, no different from how other appendices
|
||||
already reference Ch.3 equations (e.g. appendix:kde-bisection referencing method.tex
|
||||
equations).
|
||||
|
||||
Follow-up (2026-09-10, same day): user asked to add a per-sample "outcome" framing to L.1,
|
||||
calling it "the main driver" showing the practical difference between sample-wise and
|
||||
order-dependent losses. Not a new derivation -- it's the same q*/eta result from the
|
||||
existing footnote, unpacked concretely: since q*(x)>1/2 iff L(x)>eta, for ANY given x the
|
||||
classifier ends up calling it positive or negative purely by which side of eta its own
|
||||
L(x) falls on, independent of how separable x actually is. Two samples with nearly
|
||||
identical L(x) straddling eta can land on opposite sides purely because of where eta sits
|
||||
-- and since eta is fixed entirely by the chosen weight ratio, the weight choice (not the
|
||||
data) decides borderline samples' fate. Added as a new paragraph in appendix_np_theory.tex
|
||||
sec:np-class-weighting (L.1), right after the eta-derivation paragraph, before "The
|
||||
standard heuristic..." paragraph. Closing sentence makes the contrast explicit: "This is
|
||||
the central difference from an order-dependent loss: a per-sample loss must commit to some
|
||||
eta before training even begins, and every sample's outcome is filtered through that
|
||||
fixed, weight-determined cutoff." Rebuilt thesis clean (nix build exit 0), confirmed the
|
||||
new paragraph renders correctly in Appendix L.1.
|
||||
|
||||
Follow-up 2 (2026-09-10, same day): user asked why eta = (w0/w1)*(pi0/pi1) specifically had
|
||||
the pi0/pi1 factor -- explained it's NOT part of NP itself (NP just says "some eta exists,
|
||||
thresholds the likelihood ratio"); the pi0/pi1 comes from Bayes' rule converting between
|
||||
POSTERIOR odds (what a weighted loss naturally compares: w1*P(y=1|x) vs w0*P(y=0|x)) and
|
||||
LIKELIHOOD ratio (NP's native currency) -- posterior odds = likelihood ratio * prior odds,
|
||||
so translating the weighted-loss condition into likelihood-ratio form introduces pi1/pi0.
|
||||
|
||||
Follow-up 3 (2026-09-10, same day): user then asked to SIMPLIFY the whole section to avoid
|
||||
needing eta's exact formula at all -- "it exists, proportional to the class-weight ratio,
|
||||
same for all samples; changing weights changes target eta; noise changes interaction with
|
||||
eta, regardless of how eta is defined; don't need the eta=1 part, keep it simple." Realized
|
||||
the cleanest fix wasn't to hedge/soften the existing likelihood-ratio version, but to
|
||||
REWRITE L.1 entirely in POSTERIOR-ODDS terms (matching L.2, which already worked this way)
|
||||
instead of likelihood-ratio terms -- this eliminates the Bayes'-rule/prior-odds conversion
|
||||
entirely, since a weighted loss compares posteriors NATIVELY, with no translation needed:
|
||||
q*>1/2 iff O(x)=P(y=1|x)/P(y=0|x) > w0/w1, exactly, no pi0/pi1 anywhere. Kept this as an
|
||||
EXACT stated value (not hedged as merely "proportional") since the footnote genuinely
|
||||
proves the exact identity for weighted CE -- hedging it as merely proportional right next
|
||||
to a footnote proving exact equality would have been inconsistent, so instead added one
|
||||
sentence generalizing the QUALITATIVE takeaway (one constant, same for every sample, moves
|
||||
with the weight ratio) as what carries over to other per-sample losses, without claiming
|
||||
the exact w0/w1 formula does. REMOVED entirely per explicit request: the "eta=1 exactly"
|
||||
class-imbalance-cancellation paragraph (was the whole reason pi0/pi1 existed in the old
|
||||
version). Also removed L.2's opening footnote bridging its posterior-odds eta to L.1's old
|
||||
likelihood-ratio eta -- no longer needed since both sections now use the SAME O(x) object
|
||||
throughout, L.2 just directly continues from L.1's result. L.2's OWN "shrinks to w=1"
|
||||
result (the noise-validity-window boundary case) was NOT removed -- that's a different,
|
||||
still-wanted finding (about noise, not about class-imbalance cancellation), user's "don't
|
||||
need the eta=1 part" referred specifically to L.1's paragraph. Rebuilt thesis clean (nix
|
||||
build exit 0), confirmed the removed content is gone and no new undefined refs.
|
||||
|
||||
CAVEAT ADDED BY ME (not yet confirmed by user): translated the new appendix chapter title
|
||||
and the Discussion subsection's Hebrew TOC lines myself (I am not a native speaker) --
|
||||
"הצדקה תיאורטית לפונקציות הפסד ממטבות AUC לפי למת ניימן-פירסון" (appendix) and "הצדקה
|
||||
תיאורטית לפונקציות הפסד ממטבות AUC" (discussion subsection). Flagged to the user directly
|
||||
that these should be reviewed/corrected before treating this as final, unlike the rest of
|
||||
this thesis's Hebrew content which presumably had more careful authorship.
|
||||
|
||||
*** sec:np-class-weighting — class weighting as an implicit choice of eta
|
||||
|
||||
Core claim, sympy-verified before writing: for a weighted per-sample loss (weighted CE),
|
||||
minimizing pointwise risk and thresholding at q>1/2 is exactly equivalent to thresholding
|
||||
the likelihood ratio p(x|y=1)/p(x|y=0) > eta with eta = (w0/w1)*(pi0/pi1). Standard
|
||||
inverse-class-frequency weighting (w1/w0 = pi0/pi1) makes eta collapse to EXACTLY 1,
|
||||
regardless of noise/separability/distribution shape — i.e. it cancels the prior imbalance
|
||||
rather than targeting any actual desired FPR; this is a pure algebraic cancellation baked
|
||||
into the recipe (w0/w1 is DEFINED as the reciprocal of pi0/pi1), not a data-dependent
|
||||
coincidence — shown explicitly via a displayed substitution, added after the user asked why
|
||||
eta=1 held in general, not just in one example.
|
||||
|
||||
Derivation placed as a footnote (short, ~4 lines), not a full appendix — user's call
|
||||
("footnote if it's a few lines, appendicise if more"). No new bib entry added; self-contained
|
||||
derivation, not attributed to a specific citation (Elkan-style cost-sensitive learning
|
||||
literature is the likely place this already exists, but wasn't verified/added — flag for the
|
||||
user's own citation pass if wanted later).
|
||||
|
||||
Scope note (user corrected an earlier, weaker framing before this was even written into the
|
||||
thesis): an early draft claimed ROLL-AOC "doesn't commit to a single eta" by sweeping 99
|
||||
alpha values — user rejected this: at any actual decision point a classifier commits to one
|
||||
threshold, hence one implied eta; sweeping alphas during AOC training isn't the same as
|
||||
avoiding commitment to one eta at inference. Dropped; only the TPR@FPR "discovers eta as it
|
||||
goes" point was kept.
|
||||
|
||||
*** sec:np-noise-invariance — class weighting under label noise: an implicit, distorted eta
|
||||
|
||||
User's own idea (not mine), refined through two full passes:
|
||||
|
||||
PASS 1 (superseded): user proposed noise gives each sample a different "target eta."
|
||||
Checked with sympy — NOT what happens under class-conditional (x-independent) noise: the
|
||||
noisy likelihood ratio L~(x) depends on x ONLY through the true ratio L(x), via a fixed
|
||||
Mobius function g(L), strictly monotone for rho<1/2 — i.e. ONE global bijection, not
|
||||
per-sample. Written up using this density-ratio g(L) form directly.
|
||||
|
||||
PASS 2 (current thesis text): while explaining the g(L) derivation to the user, caught a
|
||||
real bug — p(x|ytilde=k) needs dividing by the normalizing constant D_k=P(ytilde=k) to be a
|
||||
proper conditional density, which the written g(L) omitted; true L~(x) = (D0/D1)*g(L(x)),
|
||||
not g(L(x)) exactly (sympy-verified). Didn't affect the monotonicity conclusion (D0/D1 is a
|
||||
positive x-independent constant) but was imprecise as written. Fixing it surfaced the real
|
||||
goal more clearly: not just "ranking is preserved" abstractly, but "noise implies an eta
|
||||
implicitly chosen by score/class weighting, one ROLL never has to choose." Reframed entirely
|
||||
in POSTERIOR-ODDS space instead of density-ratio space, which sidesteps the bug rather than
|
||||
patching it (no priors/normalizing constants needed at all): O(x)=P(y=1|x)/P(y=0|x), and
|
||||
P(ytilde=1|x) = (1-rho)P(y=1|x)+rho*P(y=0|x) via elementary total probability (no Bayes'-rule
|
||||
density manipulation, no joint-vs-conditional distinction to get wrong). Gives
|
||||
P(ytilde=1|x)/P(ytilde=0|x) = h(O(x)), h(O)=[rho+O(1-rho)]/[(1-rho)+O*rho], h'(O)=
|
||||
(1-2*rho)/[...]^2 > 0 for rho<1/2 (sympy-verified; h(O)->O at rho=0 sanity check passes).
|
||||
|
||||
Core claim: training with weight ratio w=w0/w1 on NOISY labels doesn't threshold O(x) at w
|
||||
-- it thresholds h(O(x)) at w, equivalent to thresholding the TRUE posterior odds at
|
||||
eta_true = h^{-1}(w) = [w(1-rho)-rho]/[(1-rho)-w*rho]. Whatever w you pick, you actually get
|
||||
eta_true, which depends on the (usually unknown) rho — choosing w no longer means choosing
|
||||
eta, it means choosing eta up to an unknown noise-dependent distortion. Extra result found
|
||||
and verified (2000-trial sympy/numeric check, 0 mismatches): eta_true is only a valid
|
||||
(positive) posterior odds when rho/(1-rho) < w < (1-rho)/rho; outside that window no
|
||||
classifier could produce the implied threshold at all. As rho->1/2 this window shrinks to
|
||||
just {w=1} — heavy noise leaves ONLY the no-tilt threshold achievable by weighting,
|
||||
regardless of intent. ROLL sidesteps the whole mechanism: no weight, no eta, no need to know
|
||||
rho — it targets a rate directly, and ranking by O(x) is preserved under this noise model
|
||||
(same h monotonicity fact) so the target still means what it's supposed to.
|
||||
|
||||
Caveat (user's explicit request, carried unchanged across both passes and the Ch.2->Ch.3
|
||||
move — "show it's problematic under the specific condition, doesn't mean it's untrue in
|
||||
general"): the whole argument needs rho to depend on y only (class-conditional), not x.
|
||||
Instance-dependent noise rho(x) breaks THIS ARGUMENT (h would vary with x directly) — did
|
||||
NOT claim ranking invariance actually fails there, only that this proof doesn't cover it.
|
||||
|
||||
*** Build status
|
||||
|
||||
Rebuilt thesis clean (nix build exit 0) at every step: first Ch.2 writeup, the pass-2
|
||||
rewrite, the eta=1 follow-up, and the final Ch.2->Ch.3 move. Confirmed after the move:
|
||||
sections now number 3.2.4/3.2.5 (Method chapter), \Cref{eq:roll-tpr-at-fpr} resolves to
|
||||
"(3.8)" instead of a forward-pointer, and the Ch.2 back-references (Section 2.4, Section
|
||||
2.5 for sec:imbalanced-tpr-fpr/sec:neyman-pearson) and Ch.4 forward-reference
|
||||
(\Cref{chapter:experiments} -> "Chapter 4") all resolve correctly.
|
||||
|
||||
** Gotchas
|
||||
|
||||
- KDE backward pass uses inverse-function-theorem approach deliberately; implicit differentiation
|
||||
@@ -55,3 +402,27 @@ This distinction must be stated clearly in Ch. 3 and Ch. 2 related work.
|
||||
- Beta ROLL: translation-invariance does NOT hold when scores pass through sigmoid before fitting;
|
||||
gradient-balance property breaks for Beta — documented in sec:roll-properties
|
||||
- Gradient-balance proof uses a uniform-shift argument; requires translation-invariant CDF estimator
|
||||
- eq:roll-tpr-at-fpr-gaussian and eq:gauss-grad-combined now defined in appendix_gauss_grad.tex
|
||||
(moved from method.tex) — appendix_gauss_grad.tex references both, so they must stay there
|
||||
- New .tex files must be git-added before nix build (untracked files are invisible to the
|
||||
flake's source enumeration) — bit both appendix:bandwidth-freeze and appendix:score-normalization
|
||||
on this, now routine to check
|
||||
- Never verify a derivation mentally — use sympy in the nix devShell (`nix develop --command
|
||||
python3`), and cross-check with an independent finite-difference derivative when possible
|
||||
- impl/src/roll.py's KernelizedROLLoss is a custom torch.autograd.Function with a hand-written
|
||||
backward() — it never auto-differentiates through forward()'s intermediate computations, so
|
||||
it was already implementing "frozen bandwidth" and "frozen normalization scale" by construction
|
||||
(omission), not by explicit design, until the thesis text caught up to state this as a
|
||||
deliberate convention (2026-09-09)
|
||||
- When a KEEL regression test doesn't need bit-exact old-vs-new comparison (user will say so
|
||||
explicitly, e.g. "don't need numerical exactness"), a single clean run (no NaN, sane AUC in
|
||||
the range prior runs on the same dataset/config produced) is sufficient — no need for the
|
||||
git-stash old-vs-new dance every time
|
||||
- cleveref config gotcha: \Crefname{appendix}{Appendix}{Appendices} alone does NOT make
|
||||
\Cref{appendix:...} say "Appendix" if appendices are typeset with \chapter (shares the
|
||||
"chapter" counter) — need \crefalias{chapter}{appendix} after \appendix too, otherwise every
|
||||
appendix cross-reference silently renders as "Chapter X" with no error/warning
|
||||
- Chapter boundaries aren't fixed by topic (prior work vs. original contribution) alone —
|
||||
NP the LEMMA is prior work (Ch.2), but original NP-flavored ANALYSIS the thesis contributes
|
||||
can live in Ch.3 even though it's built on Ch.2 background; don't assume "cites prior work"
|
||||
means "belongs in related work" for a whole section wholesale
|
||||
|
||||
Reference in New Issue
Block a user