model overfitting vs applicability

Question

Consider two models I built:

Model A

• I use a Neural Network to build a classification model and get a model that over fits , lets say the FPR in test set in 2 times that in train set.
• I am comfortable with the FPR on test set for my business purpose and I also check that the model test performance is reasonably stable over many other random samples of train vs Test

Model B

• I build a L1 Logistic model with a lot of features and spend a lot of time feature engineering to capture variable interactions.

• The train and test FPR are within 20% of each other for some optimal parameter, BUT the FPR here is not as good as the test FPR in Model A.

Note that the same Train and test samples are used for both models.

How much weight do I need to give to overfitting Vs prediction accuracy?

Answer 1

It is a common misconception that a model is overfit when the training and hold out error metrics are divergent. This does tend to happen when models become overfit, but it is not a sufficient condition. A better definition of overfitting is the following

A model is overfit when a (small) decrease in model complexity results in an increase in hold out performance.

Models with quite different training and hold out performance can be underfit (a small increase in complexity improves hold out performance).

So for your case, where you want all and only the model with the best hold out performance, you should focus on this aim. If you are confident in your estimate of the hold out performance of your neural network, and there are no barriers to using it in your problem domain, go with that.