I have a binary classification problem I have modeled and I'm trying to determine the best way to select my probability threshold.
Here was my modeling approach:
Create a training and testing set. My data comes in chronologically so I select data received in say Nov, Dec as my training set and Jan as my holdout (test) set.
I train the models using 5 fold cross validation and the metric I'm using is area under the precision recall curve. Based on my performance with the training set / cross validation, I then train the best model on the full training set (Nov, Dec) and predict on the hold out set (Jan).
Now I want to see the confusion matrix in the hold out set (Jan). My 'optimal' point selection on the Precision Recall Curve plotted on the hold out set (Jan) is where the F1 Score is optimized. So I select that point and look at my confusion matrix.
Now I want to use my final model. Here are my questions:
February data comes in. Do I have a 'gap' in that the final model I trained on Nov, Dec data which I tested on holdout Jan that I now use to predict classes with February data? or do I somehow incorporate Jan data (so I have Nov,Dec,Jan model) but no validation set to test on since now my model abuts February (which I want to predict)?
Assuming I use the Nov,Dec final model as the model to predict Feb, can I use the max F1 score threshold I calculated on the hold out set (Jan) when making February class predictions?
Thanks in advance.