# test validation versus k-fold cross validation

I am attempting to use a neural network, after using other machine learning algorithms. I am using the RSNNS package (I am willing to use / evaluate other packages) that's part of R. I would like to get a precision that's at least 66%.

I split the data in a training and test sets, with 4/5 of the data in the training. I then trained models using different network layouts and learning rates, using the same training set each time. I selected the parameters that gave precision >66% and the largest F-measure on the test set.

The parameters I selected gave a precision of 70% on the test set. I then took the data and did a 10-fold cross validation using the same network layout and learning rate. With this k-fold cross validation, I get a precision that is just above 50% (which is similar to the other learning algorithms I used).

My question is, is the 70% precision accurate with the test set? Is my k-fold validation possibly finding local optima, and not giving an accurate precision?

EDIT

Since this seems like an important point I left out, there are 2 classes, positive and negative. It's 18% positive, and 81% negative. There are about 550 cases.

Following Matteo's suggestion in the comments, I ran the network again multiple times. I just used the best selected parameters, because the neural network takes some time to run. I split into training (80%) and test (20%) sets again, except did 10 random splits using sample. Since it's random, some of the data appears more often in the training sets than the test sets. Using this, the precision ranged anywhere from 30% to 70%. When I averaged the 10 runs together, it came out to just above 50% precision.

I am leaning towards saying that is the best precision I can get using this data set, since the earlier machine learning algorithms gave a similar precision (data not shown).

• The precision is 70% on the test set? How is it chosen? I suggest you do to N neural network trainings with different training-testing partitions, selected randomly with sample command. – Matteo De Felice Jul 18 '14 at 16:44
• Yes, the 70% is on the test set. It was chosen randomly, using the sample command. The 10 split for the k-fold is also chosen randomly. So, should I run each network layout multiple times with different training-test sets? – treed Jul 18 '14 at 19:49
• To summarize: training 80%, testing 20% -> 70% precision (on average on how many runs?) Cross-validation: training 90%, testing 10% -> ~50% (on 10 runs) is it correct? – Matteo De Felice Jul 21 '14 at 8:44
• Does your data have a 50%/50% split on the classes? That would suggest that the final classifiers are doing a random classification. If so I could have an explanation. But let me wait for your answer. – Jacques Wainer Jul 23 '14 at 0:24
• @Mateo For the 80/20 split, there was only 1 test run, and yes, it gave 70% precision. For the cross-validation it, it gave ~50% on average over 10 runs. – treed Jul 23 '14 at 15:08

I believe there is a serious overfitting going one. When you adjust the hype-parameters using the hold out test set, you get 70% accuracy, when you run with "new" data (the 10-fold CV) you get a much lower accuracy - which is the textbook description of overfitting.

So what could be the source of the overfitting? "Old" neural networks had a big source of overfitting because of the early stop learning and thus the need for a validation set, and so on. I believe that either you are aware of this and dealt with it, or it is dealt internally in the RSNNS package

The other source of overfitting is a methodolgical one. Th protocol you used is not a (most) correct one. You did: a) a 4/5 1/5 hold out of teh whole dataset to select the hyperparameters (also known as "model selection") b) and than you did a 10-fold on the SAME dataset to evaluate the final model's accuracy.

This is not the most correct protocol. You need a nested cross validation here to select the hyperparameters and then calculate the accuracy of the selected model. I do not have the time at this moment to write about nested cross validation, but there has been some discussions and explanations about it in CV. Dikram's answer to Use of nested cross-validation is one good example (and has a link to a somewhat difficult to read paper on the topic). Internal vs external cross-validation and model selection also has great answers.

One thing that is bothering me is the magnitude of your overfitting. I have not studied this in details but in my limited experience methodological overfitting is not that high!. The only place I found this large difference in (methodological) overfitting is this one http://www.biomedcentral.com/1471-2105/7/91 but there they use artificial data where there is NO correct classifier. In that case, due to this methodological overfitting they get accuracies of 65% (average) when the true accuracy if 50%. Could this be happening in your case? There is no good classifier for your data, and the methodological error hallucinates this 70% accurate classifier!

• Thank you for your help. I knew that my protocol wasn't the best way to go about things, but thought it would be an okay shortcut to take to save some time. Apparently it was not. The links were very useful, not just for neural networks. For this project, I'm not going to continue with the neural network. – treed Jul 28 '14 at 15:47