3
$\begingroup$

I am attempting a stacked ensemble model to achieve binary classification for the first time.

Should the decision threshold be

  1. One for which I receive max F1 score (assuming F1 score drives the threshold value and not the other way round)

  2. Should be based on the business goal of the classification problem and is a question to be answered outside of the model building exercise

  3. Can be based on either sensitivity or specificity whichever is more important to the problem being solved through classification. My problem is less critical towards False Positives (to some extent) as long as there are no False Negatives.

  4. None of the above

Also considering this is a stacked ensemble, should I even be worried about the F1 scores of the base learners (each base learner algo might have a different threshold that achieves max F1 score ?)

$\endgroup$

1 Answer 1

2
$\begingroup$

There are two things to separate here:

  • The metric
  • The threshold

You should choose the metric based on business goals. If you need a good balance between precision and recall, F1 is a good choice; though as I mention in my answer to this similar question I have found models that optimize logloss tend to be more robust when released into the wild.

For the threshold, the tricky bit is assuming the best threshold for your chosen metric on your training data will be the best on your test data (or, more importantly, in production). You can plot the performance at every threshold to get a feel for how sensitive it is; the ideal is a broad flat top, as it means the choice of the threshold does not matter too much.

For models based on a time series, where e.g. train data is the older 90%, and test data is the newest 10%, I have used the average of the best threshold for each of train and test, as the value when putting the model into production. The thinking being I want to over-weight the more recent data.

For the last bit of your question, I'd treat the ensemble as a single model, when considering metric and threshold. (Though it can be tempting to make a small ensemble of models on top, that are each optimized for a different metric, or even threshold.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.