# Taking into account a non-symmetric loss function in a classification problem

1. Consider a binary classification method that estimates the class probability and where the observation weights can be specified (e.g. Logistic Regression). To accommodate the difference losses from TP and FP, which approach works better:

1a) Do not specify weights. Instead, adjust the probability threshold until the result looks good from the client's perspective.

1b) Fix the probability threshold at 0.5 and tinker the weights.

2. Consider a problem where the response has more than 2 categories and where the loss function depends on how exactly the observation is misclassified. For instance, misclassifying 1 as 5 has a bigger loss than misclassifying 1 as 2.

2a) Do I get it right that it's impossible to accommodate such loss function just by specifying observation weights?

2b) If the answer to 2a) is yes, what free software could you recommend for that problem?

2. Weighting observations won't help you if the cost of ("hard") misclassifications depends on the output class. If you weight all instances of class $$A$$ more, then your model still won't know it's better to misclassify them as $$B$$ than as $$C$$. (Of course if you play around with the weights enough, you may see an "effect" in the right direction, but this will just be noise and overfitting.)
What would have a better chance of working would, again, be probabilistic classifications. If a new instance has predicted probabilities $$\hat{p}_A$$, $$\hat{p}_B$$ and $$\hat{p}_C$$ of belonging to classes $$A$$, $$B$$ and $$C$$, then you might opt for action $$b$$ even if $$\hat{p}_C>\hat{p}_B$$ if $$\hat{p}_A$$ is "large enough", i.e., if there seems to be a high chance that the instance is actually $$A$$. Conversely, if $$\hat{p}_A$$ is small, so you are quite certain that the instance is either $$B$$ or $$C$$ (and $$C$$ is more likely because by assumption $$\hat{p}_C>\hat{p}_B$$), you might go with $$c$$ over $$b$$. (Note I am using lowercase $$a, b, c$$ for actions or decisions, rather than uppercase $$A, B, C$$ for classes, to distinguish the concepts, as above.)