2
$\begingroup$

I have data for 200 subjects and am creating a neural network to classify (binary state) for each subject. I am doing so in Matlab and have to divide up my data into training, validation, and testing sets. Originally, I created a 6-fold cross-validation method wherein I tested on a different set of 33 subjects, repeated 6 times and the accuracies averaged.

I am unsure if this is the best way to go about cross-validation or divvying up my data.

Other things I have tried include a simple 50% training, 15% validation, and 35% testing split that I repeat many times. Since I cannot guarantee that the same 35% (albeit unlikely) is selected each time, I went with the 6-fold method described above.

Now, I am trying to train/validate on all subjects aside from one, and then repeating that process 200 times for each subject.

Any ideas on what is the best/common way of dividing my data?

Improve this question
$\endgroup$
2
+50
$\begingroup$

Probably the most common method is k-fold cross validation.

Lately I've been combining k-fold cross-validation with bagging.

The algorithm is as follows:

  1. Assign folds $1:K$ to your data. For each $k$ in $K$:
  2. Select hyperparameters (weight decay, dropout, number of layers/nodes, etc.) to minimize prediction error on $k$th fold, using data not in $k$th fold.
  3. Step 2 provides $K$ networks. When predicting, average the predictions of all $K$ models. Averaging reduces the variance of the prediction.

Another motivation for this sort of averaging is that you avoid "throwing away" all of the work you did in each fold.

This can be computationally-intensive however when working with large datasets. With N = 200 however, do you really think that a neural net is the best tool for your problem?

Improve this answer
$\endgroup$
  • $\begingroup$ I tried SVM and KNN but neural networks were getting me the best accuracy. Now I'm just trying to increase the accuracy based on those initial tests $\endgroup$ – a13a22 Dec 11 '17 at 22:17
  • $\begingroup$ What about a random forest or boosted trees? $\endgroup$ – generic_user Dec 11 '17 at 22:35
  • $\begingroup$ I've tried random forests and my initial results still showed the NN to function at around 96%, while the forests and various ensembles barely crossed 85%. I'll be using NN for now, but do you see any downside to the (n-1) validation? Is the method I described of training on all but one subject the same as 200 fold cv? $\endgroup$ – a13a22 Dec 11 '17 at 23:02
  • $\begingroup$ See here re leave-one-out V's k-fold. Also read the relevant chapter of elements of statistical learning, which is free online. stats.stackexchange.com/questions/154830/… $\endgroup$ – generic_user Dec 12 '17 at 0:20
  • $\begingroup$ When you say that you got 96% from the net, was that in-sample or out of sample? $\endgroup$ – generic_user Dec 12 '17 at 0:21
0
$\begingroup$

If computational resources are not a problem, then LOOCV (leave one out cross validation) is the optimal approach. That is, training 200 models, each predicting the one left out sample, like you are already doing.

The reason for that is that cross-validation techniques generally overestimate the out-of-sample error, since they are trained on a smaller amount of samples, since you leave out a few observations. Thus, the model is undertrained with respect to all your available data, and will make less accurate predictions.

The less data you use to train the model (the larger the average hold-out-sample is), the worse your estimator.

Improve this answer
$\endgroup$
  • $\begingroup$ Is there any downside to the fact that the training set is usually the same for every test? $\endgroup$ – a13a22 Dec 14 '17 at 14:12
  • $\begingroup$ Since the training set is always the same, the 200 estimated out-of-sample errors CAN have quite a high correlation, and hence the central limit theorem does not kick in and the average can have large variance. However, in terms of bias, it's the optimal measure, and usually the variance part of it is not too bad $\endgroup$ – Sam Dec 15 '17 at 8:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.