# Using K-fold cross validation as a hint for number of neurons in hidden layer. Possible flaws?

I have a dataset of 14 continuous variables and would like to predict one (say y) using all the remaining ones (say X).

I decided to use a neural network with linear output (since I am doing regression): after a lot of reading and some tests, it turned out that a single hidden layer with about 6 or 7 neurons is a good enough configuration, however, I would like to use a better approach when selecting the neural network configuration rather than testing and testing again and again with just guessed configurations.

Say the net has a single hidden layer and we need to choose only the number of neurons.

I saw a question on Stack Overflow (which I cannot find anymore) where a user suggested a simulation where by plotting training error and testing error over the number of neurons in the hidden layer one can choose a number of neurons that minimizes the test error while keeping an eye on overfitting. I'm trying to implement this and would like to be careful not to make silly mistakes. As of now, I did the following:

test.error = NULL

for(i in 1:13)
{
# Calculate test error through cross validation
test.error[i] <- crossvalidate(data_,hidden_l=c(i))
}


The crossvalidate function takes the dataset and crossvalidates the neural network with i hidden neurons using 10-fold crossvalidation and returns the crossvalidation score (MSE).

Then by plotting the errors I get the following plot

The red line is the training error while the green one is the cross validated test error. Now my question:

Is this approach a good rule of thumb? Meaning: could I use this to get a hint of what configurations might work and then do further tests?

I'm having a hard time figuring out possible flaws since the cross validation step should rule out part of the bias due to the train-test split procedure, however I expected a more curved test error line (such as a parabola or a similar U-shaped curve such as the training error line). Do you see any flaws in this approach? Furthermore, in this example I wouldn't know how to choose between 8 and 10 neurons and can't really explain the spike at 9.

• The shape of the plot is not necessarily wrong. It is wrong to assume that cross-validation performance is smooth as a function of hyperparameters (in your case the number of neurons). For a dramatic example of this, have a look at the figure at the bottom of this page. – Marc Claesen Sep 30 '15 at 16:28
• @MarcClaesen thanks Marc, the link was useful. I think I fooled myself into this assumption due to a toy example and the example I mentioned on StackOverflow. – mickkk Sep 30 '15 at 18:51