61 lines
3.7 KiB
Org Mode
61 lines
3.7 KiB
Org Mode
:PROPERTIES:
|
||
:ID: 001430d5-e1e7-4e72-baf6-17399bfd6447
|
||
:END:
|
||
|
||
#+title: impl/loss-functions
|
||
#+filetags: :project: :knowledge: :loss-functions:
|
||
|
||
* impl/loss-functions
|
||
|
||
** KernelizedROLLoss — Key Design Decisions
|
||
|
||
- =_icdf= uses **bisection** (not Newton-Raphson). NR was replaced because =F_prime= goes near-zero when =v= is large and =tau= drifts, causing divergence. Bisection bracket is =[min(scores)−10/v, max(scores)+10/v]=, guaranteed to converge.
|
||
- Score normalisation (=normalize= flag): divides =yh= by its std before KDE. Can be disabled via =kernelized_roll_fpr(fpr, normalize=False)=. The chain-rule correction (multiply grad by =score_scale=) is applied in backward.
|
||
- =_grad_tau_scores_neg= denominator clamped at =1e-10= to prevent 0/0 when all sigmoid kernels saturate.
|
||
- Variance clamped at =1e-8= in normalisation to prevent =inf= when all batch scores are identical.
|
||
|
||
** KernelizedROLLoss Unification (2026-07-14)
|
||
|
||
=KernelizedROLLoss= and =KernelizedROLLossAOC= merged into a single class accepting =alphas= (list of CDF targets) and =divisor=:
|
||
- Point-FPR: =KernelizedROLLoss.apply([1−fpr], 1, yh, y, gamma, normalize)=
|
||
- AOC: =KernelizedROLLoss.apply(_AOC_ALPHAS, _AOC_DIVISOR, yh, y, gamma, normalize)=
|
||
|
||
Backward always vectorised (K-dim); K=1 reduces to the scalar case. =ctx.divisor= stores the non-tensor divisor.
|
||
|
||
** BCE Combination
|
||
|
||
Both =kernelized_roll_fpr= and =kernelized_roll_tpr= accept =bce_weight= (default 0.0) and =bce_pos_weight= (default 1.0).
|
||
Loss = ROLL + bce_weight × BCE. Standard value: =bce_weight=0.5=, =bce_pos_weight=num_false/num_true=.
|
||
|
||
** ISJ Bandwidth Failure
|
||
|
||
ISJ fails when a class has too few samples in a split (~<20). Affects: cleveland, ecoli, glass4, glass5.
|
||
|
||
** Gamma Scheduling
|
||
|
||
=KernelScheduler= widens KDE kernels early then anneals. Default: =initial_gamma=100=, =decay=0.5=, =decay_every=500=. Gamma is a *divisor* of =v=.
|
||
|
||
** LibAUC Integration (2026-07-14, tuned 2026-07-14)
|
||
|
||
=libauc_auc_loss= in =src/roll.py= wraps =AUCMLoss= (Yuan et al., ICCV 2021) to the =(yh, y)= convention.
|
||
- Applies =torch.sigmoid(yh)= before passing to AUCMLoss (squared-hinge surrogate, no built-in sigmoid).
|
||
- AUCMLoss has learnable params =a, b, alpha= — must use PESG optimizer (not Adam).
|
||
- =opt_factory= signature: =opt_factory(model)= — does NOT take criteriorator (would be unbound at call site in =_perform_episode=).
|
||
- =opt_factory= pattern: =loss_fn.pesg_opt_factory(...)= returns picklable =_PESGFactory= sharing the same inner =AUCMLoss= instance as the criteriorator's loss func (preserved by pickle memo).
|
||
- =ExperimentConfiguration= gained =opt_factory: callable = None=; overrides =optim_class/optim_args= when set.
|
||
- =tqdm= is an undeclared dep of libauc — must be in =propagatedBuildInputs= in flake.nix.
|
||
|
||
*** Recommended settings (v2 + adam, tuned 2026-07-14)
|
||
- =version='v2'= — removes class prior from formulation; LibAUC explicitly recommends over v1; default in =libauc_auc_loss=.
|
||
- =mode='adam'= in PESG — Adam-style primal updates, converges much faster than SGD on small datasets; default in =pesg_opt_factory=.
|
||
- =imratio= passed explicitly as =num_true / (num_true + num_false)= — avoids noisy mini-batch estimates on imbalanced data. Note: v2 ignores imratio (prior-free), so this only matters if reverting to v1.
|
||
- =lr=0.1= (PESG default, much larger than Adam's 1e-3). If diverging: try =epoch_decay=0.0= or drop lr to =0.01=.
|
||
|
||
** Grace Period (BasicCriteriorator / CRBasedCriteriorator, 2026-07-14)
|
||
|
||
=grace_period: int = 0= parameter added to both criteriorators. During the first =grace_period= epochs:
|
||
- =best_flag= is always False (no model snapshot taken)
|
||
- Patience counter does not increment (early stopping cannot fire)
|
||
|
||
All keel configs use =grace_period=50=. Prevents noisy early-epoch models from being selected as best.
|