What would be the best way of setting train/validation/test indices when dealing with classification of independent subjects? I have data for 200 subjects and am creating a neural network to classify (binary state) for each subject. I am doing so in Matlab and have to divide up my data into training, validation, and testing sets. Originally, I created a 6-fold cross-validation method wherein I tested on a different set of 33 subjects, repeated 6 times and the accuracies averaged.
I am unsure if this is the best way to go about cross-validation or divvying up my data.
Other things I have tried include a simple 50% training, 15% validation, and 35% testing split that I repeat many times. Since I cannot guarantee that the same 35% (albeit unlikely) is selected each time, I went with the 6-fold method described above.
Now, I am trying to train/validate on all subjects aside from one, and then repeating that process 200 times for each subject.
Any ideas on what is the best/common way of dividing my data?
 A: Probably the most common method is k-fold cross validation.
Lately I've been combining k-fold cross-validation with bagging.  
The algorithm is as follows:


*

*Assign folds $1:K$ to your data.  For each $k$ in $K$:

*Select hyperparameters (weight decay, dropout, number of layers/nodes, etc.) to minimize prediction error on $k$th fold, using data not in $k$th fold.  

*Step 2 provides $K$ networks.  When predicting, average the predictions of all $K$ models.  Averaging reduces the variance of the prediction.


Another motivation for this sort of averaging is that you avoid "throwing away" all of the work you did in each fold.  
This can be computationally-intensive however when working with large datasets.  With N = 200 however, do you really think that a neural net is the best tool for your problem?
A: If computational resources are not a problem, then LOOCV (leave one out cross validation) is the optimal approach. That is, training 200 models, each predicting the one left out sample, like you are already doing.
The reason for that is that cross-validation techniques generally overestimate the out-of-sample error, since they are trained on a smaller amount of samples, since you leave out a few observations. Thus, the model is undertrained with respect to all your available data, and will make less accurate predictions. 
The less data you use to train the model (the larger the average hold-out-sample is), the worse your estimator.
