I've trained a binary classification model which outputs a "probability" between (0,1).
During testing and validation, I use the ROC to measure the performance of the model. Also, I use the ROC to determine the threshold on which to cutoff false vs true predictions (e.g. I set the target to under 15% FPR).
When creating a model for production, I thought it would be ideal to train on all the available data set (e.g. no test nor validation split). Now, without test or validation split I don't have a ROC for the final model so I am without a threshold to interpret the model's output.
Is it valid to use the ROC obtained during testing? Should I calculate a new ROC of the final model over instances observed during training?
Is there something fundamentally wrong in my approach?