1
$\begingroup$

I have done some searching on the topic and I know the answer is mainly "it depends". BUT, I haven't found anything stating general relationships regarding size of data and number of hidden layers. For example, I'm working with 45-dimensional data, is there any "rule of thumb" regarding a ballpark estimate of hidden layers that may work best?

$\endgroup$
0

1 Answer 1

1
$\begingroup$

There as many "rules of thumb" as authors who wrote about this. I could tell you my personal preferences, but my answer would certainly be biased (as anyone's probably would!). There is no reason to despair tough! The good news is that, the size of the hidden layer is not so relevant. Once your choice is within a certain acceptable range, any difference in the result wouldn't be statistically significant. It's not hard to have a intuition of the reasons for this.

A neural network is universal function approximator. It draws a curve which separates the data points in different regions of space. This curve is composed by as many straight lines (hyperplanes indeed) as neurons in the hidden layer (i. e. the parameter of each hidden unit is nothing more than the parameters of a straight line). If you have too few hidden units, the curve will be composed of a small number of lines, thus it can't represent complex shapes. If you use too many hidden units, your shape can be so complex that it will model perfectly the training data, even its noisy, so it won't be able to generalize to new test cases. This is one of the most standard examples of the (in)famous bias-variance trade-off. If you have a number of hidden units that is not too big, neither too small, the curve described by you neural network will look just fine and the number of hidden units used will hardly affect the network performance.

You can find a useful set of these "rules of thumb", and also a brief discussion on the topic, in SAS FAQ.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.