I'm fitting a neural network to this example data I found online: Machine Learning Repository

I am cross validating 1 to 10 hidden units (in only 1 layer), and I have the minimum error with 10 hidden units. However, I'm somehow thinking of linearly dependent design matrices when introducing 10 hidden units for only 3 input variables (the level of Red, Green and Blue).

Is this concern justified, or can I just use 10 hidden units here? Maybe the (sigmoid) transformation does something to avoid the linear dependency?


No, there is no need to worry about this, because the non-linear transformation means that the feature space generated by the hidden layer neurons can be of higher dimension than the input space without being linearly dependent.

Consider Ripley's synthetic benchmark dataset, which consists of two classes, each of which is represented by two Gaussian clusters, which looks like this:

Ripley's synthetic benchmark datset

A good solution can be obtained by placing a radial basis function on each cluster and then using a linear discriminant on the output of these four hidden units. You should find that (even) the normal equations for linear regression are numerically well conditioned, which suggests that linear dependence isn't an issue. The non-linear transformation is indeed the reason for this.

Note that if you use regularisation (which I would recommend for any MLP application), then linear dependence isn't a problem anyway.

  • $\begingroup$ Thanks a lot! What does "MLP" stand for? $\endgroup$ – Alexander Engelhardt Aug 22 '12 at 11:15
  • $\begingroup$ no problem, MLP stands for "multi-layer perceptron", which is a common name for standard neural networks with sigmoidal (e.g. tanh or logistic) hidden layer non-linearities. $\endgroup$ – Dikran Marsupial Aug 22 '12 at 12:36
  • 1
    $\begingroup$ I'd even say that MLP refers to those neural networks which consist of layers instead of some weird directed acyclic graph structure. $\endgroup$ – bayerj Aug 22 '12 at 15:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.