Consider two models I built:
- I use a Neural Network to build a classification model and get a model that over fits , lets say the FPR in test set in 2 times that in train set.
- I am comfortable with the FPR on test set for my business purpose and I also check that the model test performance is reasonably stable over many other random samples of train vs Test
I build a L1 Logistic model with a lot of features and spend a lot of time feature engineering to capture variable interactions.
The train and test FPR are within 20% of each other for some optimal parameter, BUT the FPR here is not as good as the test FPR in Model A.
Note that the same Train and test samples are used for both models.
How much weight do I need to give to overfitting Vs prediction accuracy?