I know that there should be three sets of tests for supervised learning that are:

  • train
  • validation
  • test

I have read that for example in the case of NN in the train phase one chooses the weights and that in the validation part one should tune other factors, ie. the architecture of the NN; for example, I can choose the number of hidden layers. Also it says that I should validate the error with the validation test.

I was wondering if to apply the following set of steps is fine, for example, if I use the nnet or neuralnet package or R. These are:

  1. Obtain some datasets, for example 1000 samples, 100 samples would work as the test set.
  2. I am using a backpropagation multilayer perceptron and using the any of the packages above mentioned. I specify the inputs and the outputs of my model and run the nn.
  3. With the results from the test set, usually these packages give me the error threshold; when it is too big I start playing with the number of hidden layers or the number of neurons in the hidden layer. According to what I read this changes in the architecture should be done with the validation set, but because the error was too high with the train set I decided to tune already the number of hidden layers and neurons involved.
  4. With the adjusted parameters and with a low error presented with the training set, I start to test my model with the test set.

The results obtained with the test set, or the prediction, was really accurate with no mismatches.

Bottomline, the procedure described above is ok?, or should I tune the weights with the training set and then with other set tune the hidden layers, number of neurons and so on.

  • $\begingroup$ So (1) you have a training set of 1000 samples, a test set of 100 samples, and no validation set, is it right? (2) Are your training set and your test set intersect (contain common observations)? (3) You always train your neural net on the training set only, is it right? (4) You alter the metaparameters (number of hidden layers, number of neurons in each) so as to obtain the best possible error (4a) on the training set? or (4b) on the test set? (5) Had you do the cross-validation on some stage? $\endgroup$ – user31264 Mar 28 '15 at 7:14
  • 2
    $\begingroup$ Possible duplicate of How can I help ensure testing data does not leak into training data? $\endgroup$ – Sycorax Aug 15 '18 at 23:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.