# Linear regression with redundant features (perfect multicolinearity)

Suppose $X \sim N(0,1)$, $Z=X$, and $Y=X$. An ordinary least squares regression problem is solved:

$min_{(b1,b2)} \|Y-(b1*X+b_2*Z)\|_{2}^2$

This is a strictly convex function which must have a unique solution. But this example has three solutions: $(1,0), (0,1), (0.5,0.5)$. How can a strictly convex function have more than one solution?

Also, why do convex solvers prefer $(0.5,0.5)$?

• it has an infinite number of solutions of the form $(a,1-a)$, not merely 3. – Glen_b Nov 23 '14 at 9:32
• What is your basis for claiming "this is a strictly convex function"? Exactly what function are you referring to? – whuber Nov 23 '14 at 21:07

By the identity assumptions stated in the question

$$Y-(b_1X+b_2Z) = X - b_1X-b_2X = (1-b_1-b_2)X$$

and so

$$Q\equiv \|(1-b_1-b_2)X\|_{2}^2 = (1-b_1-b_2)^2\sum_{i=1}^nx_i^2$$

Its minimum is $0$, obtained for all pairs $$(b_1^*, b_2^*): b_1^*+ b_2^*=1$$

as Glen_b noted in a comment.

Consider the Hessian: $$\frac{\partial Q}{\partial b_1}=\frac{\partial Q}{\partial b_2} =-(1-b_1-b_2)\sum_{i=1}^nx_i^2$$

and so

$$\frac{\partial^2 Q}{\partial b_1^2} = \frac{\partial^2 Q}{\partial b_2^2}= \frac{\partial^2 Q}{\partial b_1\partial b_2}=\sum_{i=1}^nx_i^2$$.

So the determinant of the Hessian is zero.
The function does have a global minimum value ($0$), but the minimizer vector is not unique. This happens because due to the equality constraints, the values of the $X$'s do not affect the solution.