1. Consider a binary classification method that estimates the class probability and where the observation weights can be specified (e.g. Logistic Regression). To accommodate the difference losses from TP and FP, which approach works better:

    1a) Do not specify weights. Instead, adjust the probability threshold until the result looks good from the client's perspective.

    1b) Fix the probability threshold at 0.5 and tinker the weights.

  2. Consider a problem where the response has more than 2 categories and where the loss function depends on how exactly the observation is misclassified. For instance, misclassifying 1 as 5 has a bigger loss than misclassifying 1 as 2.

    2a) Do I get it right that it's impossible to accommodate such loss function just by specifying observation weights?

    2b) If the answer to 2a) is yes, what free software could you recommend for that problem?


1 Answer 1

  1. I would recommend separating the modeling and the decision aspect: model probabilistically, assess these probabilistic classifications using proper scoring rules, and use thresholds when turning these classifications into actions or decisions. There may be multiple possible actions even if there are only two classes, which multiple thresholds can accommodate: Reduce Classification Probability Threshold

    I would not try to bias class probabilities just to be able to work with a 0.5 threshold afterwards. This is pretty much the same as oversampling, Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?, and feels like cutting a groove into the head of your nail so you can use a screwdriver to get it into the wall instead of reaching for your hammer.

  2. Weighting observations won't help you if the cost of ("hard") misclassifications depends on the output class. If you weight all instances of class $A$ more, then your model still won't know it's better to misclassify them as $B$ than as $C$. (Of course if you play around with the weights enough, you may see an "effect" in the right direction, but this will just be noise and overfitting.)

    What would have a better chance of working would, again, be probabilistic classifications. If a new instance has predicted probabilities $\hat{p}_A$, $\hat{p}_B$ and $\hat{p}_C$ of belonging to classes $A$, $B$ and $C$, then you might opt for action $b$ even if $\hat{p}_C>\hat{p}_B$ if $\hat{p}_A$ is "large enough", i.e., if there seems to be a high chance that the instance is actually $A$. Conversely, if $\hat{p}_A$ is small, so you are quite certain that the instance is either $B$ or $C$ (and $C$ is more likely because by assumption $\hat{p}_C>\hat{p}_B$), you might go with $c$ over $b$. (Note I am using lowercase $a, b, c$ for actions or decisions, rather than uppercase $A, B, C$ for classes, to distinguish the concepts, as above.)

  • $\begingroup$ Hi Stephan, thanks a lot for replying. I found Harrell article, it's very helpful. $\endgroup$
    – James
    Commented Jun 13 at 18:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.