Feature correlation and their effect of Logistic Regression I was reading this nice post by Jason Brownlee.
In the post there is a paragraph: 

This is useful to know, because some machine learning algorithms like
  linear and logistic regression can have poor performance if there are
  highly correlated input variables in your data.

It is not clear to me why this is true. Assuming that we have enough instances to train on, then this should not be a learning problem. Any technical explanation is appreciated. 
 A: I'll try to give an intuitive answer, without getting too technical:
In the extreme case, consider some variables that are perfectly correlated, i.e. linearly dependent.
Let's say you're using logistic regression to predict the probability of a car accident being fatal, where one of the predictor variables will be speed, but for some reason you read in two input variables $x$ = speed in km/h and $y$ = speed in mph.
Clearly, $x$ and $y$ are linearly dependent as $x = 1.61y$.
In your model, you will estimate coefficients $\beta$ such that
$$log\left(\frac{P(fatal)}{P(not fatal)}\right) = \beta_0 + \beta_x x + \beta_y y + (other \ terms) $$
Due to the linear dependence, the relevant term on the right hand side can be rewritten as $(1.61\beta_x + \beta_y)y$. Now let's say that for some reason the speed actually has no influence on the fatality of the crash (obviously not true in real life). Then you would expect your model to find $\beta_x = \beta_y = 0$.
However, due to linear dependence, the model cannot distinguish between $\beta_x = \beta_y = 0$ or $\beta_x = 1, \beta_y = -1.61$, or $\beta_x = -100, \beta_y = 161$ or any other combination where $\beta_x = - \frac{\beta_y}{1.61}$.
Thus on the one hand, your model itself might become numerically unstable. On the other hand, you can no longer interpret the results correctly.
The same is true when variables are no longer linearly dependent but highly correlated (Think speed measurements in the same unit but from two distinct systems inside the car, such as $x$ = speed in km/h measured by counting wheel rotations per second and $y$ = speed in km/h measured using GPS. Then due to measurement errors, we expect $x$ and $y$ to not be exactly the same, but very highly correlated. Let's keep the assumption that the true coefficients should be 0. Then again, when looking at the entire input data set, the model can hardly distinguish between any cases where $\beta_x = - \beta_y$.
As such correlation can lead to poor performance of regression (and other) ML algorithms.
A: Consider linear regression. The solution is given by
$$
\hat\beta = (X^T X)^{-1} X^T y
$$
where each row of X is an observation, and each column a feature. It holds that
$$
\operatorname{rank} X^T X = \operatorname{rank} X .
$$
So if some two features are linearly dependent, then $X$ and $X^T X$ are rank deficient, and $X^T X$ cannot be inverted. If the features are not completely dependent but highly correlated, $X^T X$ would be nearly rank deficient, and inversion would be numerically unnstable.
