Threshold Tuning before or after parameter tuning?

Question

My goal is to increase the F1 score of Class 1 by 1-2%.

I achieved this by changing the threshold from 0.5 to X using the precision recall curve when the dataset is imbalanced. I did this after I have done with the parameter tuning (ie First, parameter tuning. Then, threshold tuning)

However, my senior told me that I should do threshold tuning before the parameter tuning.

The problem is that:

1. I don't know how to do threshold tuning in Tensorflow Keras before the parameter tuning (how many hidden layers, how many neurons, do i use weights, do i use biases).
2. My senior did not give proof why I should do Threshold Tuning before Parameter Tuning.

Any help to explain why I should do Threshold Tuning before Parameter Tuning would be greatly appreciated. I just need the motivation to read more docs about the Tensorflow Keras.

Answer 1

You shouldn't. This advice is not helpful. (Unless there is some context we don't know.)

First, your precision-recall curve will depend on your model, i.e., on your estimated parameters. A different model will give rise to a different precision-recall curve, so the optimal threshold to give the best F1 score will be different. Tuning the threshold before you even have the precision-recall curve (how would you even do that?) is putting the cart before the horse.

Second, and more generally, any "optimal" threshold should not be part of the statistical model, because it does not only depend on the model, but also on the costs of classifications, decisions or actions. See here, and links therein.

Third, you may answer my second point by pointing to the fact that optimizing the F1 score does provide us with a tradeoff of costs. It doesn't, or to be more precise, it yields one very specific cost tradeoff and therefore keeps us from thinking about the cost structure. (It does not even address the issue of quite possibly having more than two possible actions, even if we have only two classes, see my answer linked to above.) This creates a bias towards positive or negative classifications, just as accuracy and all related KPIs like precision, recall etc., which all suffer from the same major problems.

The solution is to stay with the probabilistic classifications your model already outputs, which together with varying thresholds make up your precision-recall curve. Just don't apply thresholds to them yet. Instead, evaluate them using proper scoring rules. Once you have found a well-calibrated model, think about the cost structure, and then use the probabilistic predictions from your model together with that cost structure to derive optimal actions using one or multiple thresholds.

You may want to point your senior to this thread. If they disagree with this argument, I would be interested in why.