test validation versus k-fold cross validation

Question

I am attempting to use a neural network, after using other machine learning algorithms. I am using the RSNNS package (I am willing to use / evaluate other packages) that's part of R. I would like to get a precision that's at least 66%.

I split the data in a training and test sets, with 4/5 of the data in the training. I then trained models using different network layouts and learning rates, using the same training set each time. I selected the parameters that gave precision >66% and the largest F-measure on the test set.

The parameters I selected gave a precision of 70% on the test set. I then took the data and did a 10-fold cross validation using the same network layout and learning rate. With this k-fold cross validation, I get a precision that is just above 50% (which is similar to the other learning algorithms I used).

My question is, is the 70% precision accurate with the test set? Is my k-fold validation possibly finding local optima, and not giving an accurate precision?

EDIT

Since this seems like an important point I left out, there are 2 classes, positive and negative. It's 18% positive, and 81% negative. There are about 550 cases.

Following Matteo's suggestion in the comments, I ran the network again multiple times. I just used the best selected parameters, because the neural network takes some time to run. I split into training (80%) and test (20%) sets again, except did 10 random splits using sample. Since it's random, some of the data appears more often in the training sets than the test sets. Using this, the precision ranged anywhere from 30% to 70%. When I averaged the 10 runs together, it came out to just above 50% precision.

I am leaning towards saying that is the best precision I can get using this data set, since the earlier machine learning algorithms gave a similar precision (data not shown).

Answer 1

I believe there is a serious overfitting going one. When you adjust the hype-parameters using the hold out test set, you get 70% accuracy, when you run with "new" data (the 10-fold CV) you get a much lower accuracy - which is the textbook description of overfitting.

So what could be the source of the overfitting? "Old" neural networks had a big source of overfitting because of the early stop learning and thus the need for a validation set, and so on. I believe that either you are aware of this and dealt with it, or it is dealt internally in the RSNNS package

The other source of overfitting is a methodolgical one. Th protocol you used is not a (most) correct one. You did: a) a 4/5 1/5 hold out of teh whole dataset to select the hyperparameters (also known as "model selection") b) and than you did a 10-fold on the SAME dataset to evaluate the final model's accuracy.

This is not the most correct protocol. You need a nested cross validation here to select the hyperparameters and then calculate the accuracy of the selected model. I do not have the time at this moment to write about nested cross validation, but there has been some discussions and explanations about it in CV. Dikram's answer to Use of nested cross-validation is one good example (and has a link to a somewhat difficult to read paper on the topic). Internal vs external cross-validation and model selection also has great answers.

One thing that is bothering me is the magnitude of your overfitting. I have not studied this in details but in my limited experience methodological overfitting is not that high!. The only place I found this large difference in (methodological) overfitting is this one http://www.biomedcentral.com/1471-2105/7/91 but there they use artificial data where there is NO correct classifier. In that case, due to this methodological overfitting they get accuracies of 65% (average) when the true accuracy if 50%. Could this be happening in your case? There is no good classifier for your data, and the methodological error hallucinates this 70% accurate classifier!