I'm fitting a neural network to this example data I found online: Machine Learning Repository
I am cross validating 1 to 10 hidden units (in only 1 layer), and I have the minimum error with 10 hidden units. However, I'm somehow thinking of linearly dependent design matrices when introducing 10 hidden units for only 3 input variables (the level of Red, Green and Blue).
Is this concern justified, or can I just use 10 hidden units here? Maybe the (sigmoid) transformation does something to avoid the linear dependency?