1
$\begingroup$

I got a problem in which the classification task is very hard to accomplish, because the features are not very informative.

Anyway I'm trying to get some results (even poor) using a neural network (MLP) tuning its hyper parameters. Because I'm searching for very little improvement for the classifier and because the problem is very hard, I'm expecting that in the training set, the training error will be not too low (accuracy not too high).

So I thought of a "reliability" measure, which is in fact a similarity measure between the train and test performance:

sim = 1 - |train_err - test_err|

If train and test are near (sim ~ 1) , it means that the training is reliable (no overfitting), instead if train and test are far (sim << 1) this means that the network has probaby overfitted the training set.

Makes it sense to use this metric as "reliability" measure for identify overfitting?

$\endgroup$
1
$\begingroup$

With this definition, the similarity increases with the sample size. For example, in the limit where the training data are half of the population and the test data are the other half, Sim is almost 1. To make this a statement about the method and not about the pair (method, sample), you need to adjust for the sample size.

$\endgroup$
  • $\begingroup$ What do you mean as "Adjust for the sample size"? Do you mean something like a normalization? $\endgroup$ – Nikaidoh Mar 23 '18 at 11:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.