Does correlated input data lead to overfitting with neural networks?

Question

In my opinion correlated input data must lead to overfitting in neural networks because the network learns the correlation e.g. noise in the data.

Is this correct?

Answer 1

Actually no.

The question as such is a bit general, and mixes two things that are not really related. Overfitting usually is meant as the opposing quality to being a generalized description; in the sense that an overfitted (or overtrained) network will have less generalization power. This quality is primarily determined by the network architecture, the training and the validation procedure. The data and its properties only enter as "something that the training procedure happens on". This is more or less "text book knowledge"; you could try "An Introduction to Statistical Learning" by James, Witten, Hastie and Tibshirani. Or "Pattern Recognition" by Bishop (my favourite book ever on the general topic). Or "Pattern Recognition and Machine Learning", also by Bishop.

For the correlation itself: Consider the input space having a certain dimension. No matter what transformation you use, the dimensionality will remain the same -- linear algebra says so. In one case the given base will be completely uncorrelated -- this is what you get, when you de-correlate the variables, or simply apply PAT (Principle Axis Transformation.) Take any linear algebra book for this.

Since a neural network with an appropriate architecture can model any (!) function, you can safely assume, that it also could first model the PAT and then do whatever it also should do -- e.g. classification, regression, etc.

You could also consider the correlation a feature, which should be part of the neural network description, since it's a property of the data. The nature of the correlation is not really important, unless it is something that should not be a part of the data. This would actually be a different topic -- you should model or quantify something like noise in the input and account for it.

So, in summary no. Correlated data means you should work harder to make the handling of data technically simpler and more effective. Overfitting can occur, but in won't happen because there is correlated data.

Answer 2

cherub is correct in regards to his statement's pertaining to over-fitting. However, I think the discussion of highly correlated features and ANN overly simplifies the issue.

Yes, it is true in theory that an ANN can approximate any function. However, in practice is it not a good idea to include numerous highly correlated features. Doing so will introduce many redundancies within the model. The inclusion of such redundancies will introduce unnecessary complexities and in doing so could increase the number of local minima. Given that the loss function of an ANN is not inherently smooth, introducing unnecessary roughness is not a great idea.

Answer 3

Well, AlphaGo's value network has suffered from correlated data according to the authors, David Silver et al.. The network predicts the outcome of a game, and the training data consists of professional games. They say that the network learned to recognize the games in the training data instead of learning the game itself. This is very plausible, learning the game is hard, but recognizing distinct games are much easier, every game probably has some easily recognizable characteristic features. To mitigate this problem, they created a new dataset by taking only a single state from a game. This reduced the overfitting and the network generalized.

References:

Silver, David, et al. "Mastering the game of Go with deep neural networks and tree search." nature 529.7587 (2016): 484-489.