test validation versus k-fold cross validation

I am attempting to use a neural network, after using other machine learning algorithms. I am using the RSNNS package (I am willing to use / evaluate other packages) that's part of R. I would like to get a precision that's at least 66%.

I split the data in a training and test sets, with 4/5 of the data in the training. I then trained models using different network layouts and learning rates, using the same training set each time. I selected the parameters that gave precision >66% and the largest F-measure on the test set.

The parameters I selected gave a precision of 70% on the test set. I then took the data and did a 10-fold cross validation using the same network layout and learning rate. With this k-fold cross validation, I get a precision that is just above 50% (which is similar to the other learning algorithms I used).

My question is, is the 70% precision accurate with the test set? Is my k-fold validation possibly finding local optima, and not giving an accurate precision?

EDIT

Since this seems like an important point I left out, there are 2 classes, positive and negative. It's 18% positive, and 81% negative. There are about 550 cases.

Following Matteo's suggestion in the comments, I ran the network again multiple times. I just used the best selected parameters, because the neural network takes some time to run. I split into training (80%) and test (20%) sets again, except did 10 random splits using sample. Since it's random, some of the data appears more often in the training sets than the test sets. Using this, the precision ranged anywhere from 30% to 70%. When I averaged the 10 runs together, it came out to just above 50% precision.

I am leaning towards saying that is the best precision I can get using this data set, since the earlier machine learning algorithms gave a similar precision (data not shown).

• The precision is 70% on the test set? How is it chosen? I suggest you do to N neural network trainings with different training-testing partitions, selected randomly with sample command. – Matteo De Felice Jul 18 '14 at 16:44
• Yes, the 70% is on the test set. It was chosen randomly, using the sample command. The 10 split for the k-fold is also chosen randomly. So, should I run each network layout multiple times with different training-test sets? – treed Jul 18 '14 at 19:49
• To summarize: training 80%, testing 20% -> 70% precision (on average on how many runs?) Cross-validation: training 90%, testing 10% -> ~50% (on 10 runs) is it correct? – Matteo De Felice Jul 21 '14 at 8:44
• Does your data have a 50%/50% split on the classes? That would suggest that the final classifiers are doing a random classification. If so I could have an explanation. But let me wait for your answer. – Jacques Wainer Jul 23 '14 at 0:24
• @Mateo For the 80/20 split, there was only 1 test run, and yes, it gave 70% precision. For the cross-validation it, it gave ~50% on average over 10 runs. – treed Jul 23 '14 at 15:08