This is more of a general question. But, when you feed in inputs into a machine learning algorithm, are the inputs typically dependent or dependent? What are the implications if the inputs are independent or dependent?


Consider an input space $D$, so that the inputs to the ML algorithm are $\{x_1,\ldots,x_m\}$ for each $x_i\in D$.

Normally, the separate inputs $x_j$ are assumed to be independent (and identically distributed). This property tends to be assumed in PAC bounds, for instance. Of course, elements of a dataset are not exactly independent in reality, but this is not usually considered a huge issue in ML (or rather it tends to be ignored) except in cases where it severely impacts performance (like in reinforcement learning: replay is used to help enforce sample independence, for instance).

But what about correlations among input features, i.e. within a single input? This question requires thinking about the data itself.

  • Consider $D = \mathbb{R}^d$, so $x_k$ is just a vector. The other answer discusses this case, and I don't have much to add. Essentially, dependence between features harms the algorithm. Two features with high correlation, for instance, contain little new information, but may add additional noise and can over-weight the same information (meaning the model has to learn to ignore some of it). This is the motivation for the famous minimal-redundancy-maximal-relevance feature selection strategy.

  • Consider sequence information, such as sentences, so e.g. $D=\mathbb{R}^{T\times n}$. Here we need to consider temporal dependencies, which are often very important. (The end of a sentence does indeed depend on the beginning of a sentence!) So notice that the model does indeed need to learn to handle correlations across the time ($T$) dimension. In other words, for e.g. sentences, there are known to be strong temporal dependencies: it can be useful to take advantage of this structure in our models. This is part of the motivation for using RNNs, in cases where the dependency structure is somewhat directional. Also, we again face the same issues as in the first case, with the vectors per member of each sequence (i.e. between features, across the $n$ dimension) potentially being dependent, which is generally not good.

  • Consider image features, so $D = \mathbb{R}^{W\times H\times C}$. In this case, spatial dependencies are extremely structured and powerful (i.e. across $W$ and $H$). Intra-channel dependencies are usually not so important. This fact is exactly why convolutional networks work so well: they exploit the local spatial correlations via using small convolutional kernels.

Ok so overall:

But, when you feed in inputs into a machine learning algorithm, are the inputs typically dependent or dependent?

Between training set samples, they are assumed to be independent. Theoretically, this is ideal. They can be close to this or far from it in practice. Between features and/or within a single input, dependencies can be very strong and structured.

What are the implications if the inputs are independent or dependent?

It depends. If we know something about the dependency (as in computer vision and natural language processing), then we can use it to design better models (e.g., CNNs). If we don't, then dependencies are generally bad. For example, a special case of this is an unknown multicollinearity, which implies a linear dependency structure - this is damaging to simple predictors in general. Deep neural networks, however, are often assumed to be capable of learning and then ignoring unwanted feature dependencies (or exploiting them).


The subject you are referring to is called Multicollinearity.

When some features of your input data are correlated, it might decrease your model performance. Nevertheless, the impact highly depends on your data and the type of model you are using. You can read more about it here:

Multicollinearity happens when one predictor variable in a multiple regression model can be linearly predicted from the others with a high degree of accuracy. This can lead to skewed or misleading results. Luckily, decision trees and boosted trees algorithms are immune to multicollinearity by nature. When they decide to split, the tree will choose only one of the perfectly correlated features. However, other algorithms like Logistic Regression or Linear Regression are not immune to that problem and you should fix it before training the model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.