I got a problem in which the classification task is very hard to accomplish, because the features are not very informative.
Anyway I'm trying to get some results (even poor) using a neural network (MLP) tuning its hyper parameters. Because I'm searching for very little improvement for the classifier and because the problem is very hard, I'm expecting that in the training set, the training error will be not too low (accuracy not too high).
So I thought of a "reliability" measure, which is in fact a similarity measure between the train and test performance:
sim = 1 - |train_err - test_err|
If train and test are near (sim ~ 1) , it means that the training is reliable (no overfitting), instead if train and test are far (sim << 1) this means that the network has probaby overfitted the training set.
Makes it sense to use this metric as "reliability" measure for identify overfitting?