1
$\begingroup$

For binary classification problems, how does reweighting the positive/negative classes affect the learned classifier?

In the case of 0-1 loss, it’s clear that this is equivalent to trading off FNR for FPR. I think the case of the logistic loss is a bit more interesting: since the Bayes classifier is $\log \eta(x)/(1-\eta(x))$, reweighting terms is equivalent to adding a constant to the ranker. This sounds like the 0-1 loss case where the constant factor moves the threshold for your classifier away from $1/2$.

For the exponential loss we can say a bit more: consider a classifier that can be expressed as $g(x) + t$, we can show that weighting the positive and negative classes is equivalent to moving the threshold up or down:

First, we can find the optimal t by minimizing $$\mathcal L(g + t) = \sum w(y_i) e^{ -y_i (g(x_i) + t) }$$ Substituting the optimal $t$ into the loss gives an objective that is independent of the weights $w$. So applying a weighting is equivalent to choosing a threshold for our classifier.

I’m struggling to show a similar result (or disprove it) for the logistic loss when the Bayes classifier is not attainable. For example, if I’m selecting from the class $\{\theta^T x | \theta \in R^d\}$ it’s not clear to me that weighting the loss terms is always equivalent to selecting the classification threshold. Optimizing $t$ first like we did for the exponential loss doesnt seem to work out in closed form here. Maybe we can show the same result holds approximately?

$\endgroup$

1 Answer 1

1
+200
$\begingroup$

It's only approximately true in general. The relevant theory is that of case-control sampling, from biostatistics. Scott & Wild, JRSS B, 2002 is a good reference, though they don't phrase the question in a way that looks like your question.

Suppose that $$\mathrm{logit} P[Y=1|X=x] = \alpha+\beta^Tx$$ and you split at 0.5 for turning probabilities into predictions.

If that's true, weighting by weights $w_Y$ gives asymptotically the same classifier as changing the threshold by $\log (w_1/w_0)$, in the sense that the estimators of $\beta$ converge to the same limits. Changing the weights is less statistically efficient -- the sampling uncertainty in $\hat\beta$ will be larger for the weighted estimator -- but the sampling uncertainty is usually asymptotically negligible compared to the Bayes risk.

If it's not true that $$\mathrm{logit} P[Y=1|X=x] = \alpha+\beta^Tx,$$ eg, if you make the right-hand side linear in $x_{13}$ and it's actually non-linear, the weighted estimator and the changed-threshold estimator do not converge to the same thing. The approximation is usually pretty good unless the model is grossly wrong, which it probably isn't.

So, in practice, weighted estimation and changing the threshold will be close for the logistic loss, but not numerically identical. The exception is a saturated model, where $x$ is a single discrete variable or a complete cross-classification of a set of discrete variables, where they will be identical -- basically, because there's only one estimator in that setting.

$\endgroup$
2
  • $\begingroup$ Thanks for the link, I found the discussion in section 2.1 to be interesting. There is a comment in there "By contrast, if the model is misspecified, every single element of $B_\lambda$ depends on the weightings that are used". This is the claim that I am interested in proving. Do you have any references for this statement? $\endgroup$
    – dmh
    Jan 5, 2021 at 20:31
  • 1
    $\begingroup$ The rest of section 2.1 is a derivation for small departures from correct model specification -- they compute the first-order difference between the two. For large departures I don't know of a proof that they are never the same, but they must be different almost everywhere. $\endgroup$ Jan 5, 2021 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.