13
$\begingroup$

In my opinion correlated input data must lead to overfitting in neural networks because the network learns the correlation e.g. noise in the data.

Is this correct?

$\endgroup$
11
+50
$\begingroup$

Actually no.

The question as such is a bit general, and mixes two things that are not really related. Overfitting usually is meant as the opposing quality to being a generalized description; in the sense that an overfitted (or overtrained) network will have less generalization power. This quality is primarily determined by the network architecture, the training and the validation procedure. The data and its properties only enter as "something that the training procedure happens on". This is more or less "text book knowledge"; you could try "An Introduction to Statistical Learning" by James, Witten, Hastie and Tibshirani. Or "Pattern Recognition" by Bishop (my favourite book ever on the general topic). Or "Pattern Recognition and Machine Learning", also by Bishop.

For the correlation itself: Consider the input space having a certain dimension. No matter what transformation you use, the dimensionality will remain the same -- linear algebra says so. In one case the given base will be completely uncorrelated -- this is what you get, when you de-correlate the variables, or simply apply PAT (Principle Axis Transformation.) Take any linear algebra book for this.

Since a neural network with an appropriate architecture can model any (!) function, you can safely assume, that it also could first model the PAT and then do whatever it also should do -- e.g. classification, regression, etc.

You could also consider the correlation a feature, which should be part of the neural network description, since it's a property of the data. The nature of the correlation is not really important, unless it is something that should not be a part of the data. This would actually be a different topic -- you should model or quantify something like noise in the input and account for it.

So, in summary no. Correlated data means you should work harder to make the handling of data technically simpler and more effective. Overfitting can occur, but in won't happen because there is correlated data.

$\endgroup$
  • $\begingroup$ Sorry, but I still didn't get why. Also my doubt is a bit generic. I was looking for an answer to "Can correlated input data be harmful for neural networks?". Here you claim, "you can safely assume that it also could first model PAT". But how are you making that assumption. And then my followup question is what happens if the neural architecture doesn't for fail to model PAT? $\endgroup$ – bytestorm Aug 11 '19 at 17:37
  • $\begingroup$ @bytestorm: your first question is different than the original. Correlated input might limit the performance of your ANN (as well as for other methods). But it is not an inherent property of an ANN. As for the second question, it is less an assumption, but merely an exemplary explanation why ANNs could model PAT. In practice I wouldn't do it like that, nor recommend to do that. $\endgroup$ – cherub Aug 14 '19 at 14:06
6
$\begingroup$

cherub is correct in regards to his statement's pertaining to over-fitting. However, I think the discussion of highly correlated features and ANN overly simplifies the issue.

Yes, it is true in theory that an ANN can approximate any function. However, in practice is it not a good idea to include numerous highly correlated features. Doing so will introduce many redundancies within the model. The inclusion of such redundancies will introduce unnecessary complexities and in doing so could increase the number of local minima. Given that the loss function of an ANN is not inherently smooth, introducing unnecessary roughness is not a great idea.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.