Having trouble with hidden layer neurons in neural network

Question

I have some doubts related to my neural network implementation. I have 750 features and 98 outputs. I have number of samples = 5000. Now I used cross validation to select the number of neurons in the hidden layer(1 hidden layer). I used 10 fold cross validation. I got that number of neurons = 30 performed the best which is kind of strange. I couldn't explain what's going on here. Any ideas/ suggestions?

How could just 30 neurons do better with such high dimensional data

I used tansig activation for the hidden layer and logsig for the output. The outputs used for training were scaled to the range of [0 1]. The inputs where standardized to zero mean and unit variance.

Answer 1

The short version is that the number of Neurons in the hidden layer does not necessarily have an easy numerical relationship to the number of features. High-dimensional data sets can sometimes be explained or modeled using relatively few relationships; similarly, low dimensional data can sometime require a shockingly rich model in order to model the data sufficiently well.

Consider intuitively the case where there are many dimensions, but many of them are irrelevant, noisy, or otherwise not useful for the modelling task (you can imagine doing something like factor analysis to determine this). In this case, we need to model relationships that are relatively few in number compared to the number of inputs, so we only need a few neurons in the hidden layer.

Conversely, we can have a situation where relatively low dimensional data might require many hidden neurons; the output can be some very, very complicated function of the inputs, and so we are forced to use a very rich model to explain the relationships between the variables even if the number of inputs is quite small.

Unfortunately, the takeaway is that the particular choice of number of hidden neurons is not particularly easy; however, cross validation (while possibly expensive) is a relatively reliable choice for choosing the number of neurons in the hidden layer.

Answer 2

The biggest problem with simple, classical neural network is their overfitting capabilities. If you do not use any type of regularization (like eg. Tikhonov regularization), then the only parameter which prevents overfitting is small number of hidden neurons. Assuming, that you dit not make any implementation error, it looks like ~30 is in your case kind of equlibrium between expression power (larger hidden layer) and model simplicity and so-smaller chance of overfitting (smaller hidden layer).

Smaller number of hidden neurons then output ones also suggest high correlation of your output values (as well as input ones).

To sum up:

• you should double check the code to be sure that everything is ok
• if it is ok, then there are at least two possible reasons:
  • there is a high correlation in input/output values of your data
  • found size of the layer is kind of equlibrium between model's expresivness and complexity