I know that there should be three sets of tests for supervised learning that are:
I have read that for example in the case of NN in the train phase one chooses the weights and that in the validation part one should tune other factors, ie. the architecture of the NN; for example, I can choose the number of hidden layers. Also it says that I should validate the error with the validation test.
I was wondering if to apply the following set of steps is fine, for example, if I use the nnet or neuralnet package or R. These are:
- Obtain some datasets, for example 1000 samples, 100 samples would work as the test set.
- I am using a backpropagation multilayer perceptron and using the any of the packages above mentioned. I specify the inputs and the outputs of my model and run the nn.
- With the results from the test set, usually these packages give me the error threshold; when it is too big I start playing with the number of hidden layers or the number of neurons in the hidden layer. According to what I read this changes in the architecture should be done with the validation set, but because the error was too high with the train set I decided to tune already the number of hidden layers and neurons involved.
- With the adjusted parameters and with a low error presented with the training set, I start to test my model with the test set.
The results obtained with the test set, or the prediction, was really accurate with no mismatches.
Bottomline, the procedure described above is ok?, or should I tune the weights with the training set and then with other set tune the hidden layers, number of neurons and so on.