Suppose $X \sim N(0,1)$, $Z=X$, and $Y=X$. An ordinary least squares regression problem is solved:

$min_{(b1,b2)} \|Y-(b1*X+b_2*Z)\|_{2}^2$

This is a strictly convex function which must have a unique solution. But this example has three solutions: $(1,0), (0,1), (0.5,0.5)$. How can a strictly convex function have more than one solution?

Also, why do convex solvers prefer $(0.5,0.5)$?

  • $\begingroup$ it has an infinite number of solutions of the form $(a,1-a)$, not merely 3. $\endgroup$ – Glen_b Nov 23 '14 at 9:32
  • $\begingroup$ What is your basis for claiming "this is a strictly convex function"? Exactly what function are you referring to? $\endgroup$ – whuber Nov 23 '14 at 21:07

By the identity assumptions stated in the question

$$Y-(b_1X+b_2Z) = X - b_1X-b_2X = (1-b_1-b_2)X$$

and so

$$Q\equiv \|(1-b_1-b_2)X\|_{2}^2 = (1-b_1-b_2)^2\sum_{i=1}^nx_i^2$$

Its minimum is $0$, obtained for all pairs $$(b_1^*, b_2^*): b_1^*+ b_2^*=1$$

as Glen_b noted in a comment.

Consider the Hessian: $$\frac{\partial Q}{\partial b_1}=\frac{\partial Q}{\partial b_2} =-(1-b_1-b_2)\sum_{i=1}^nx_i^2$$

and so

$$\frac{\partial^2 Q}{\partial b_1^2} = \frac{\partial^2 Q}{\partial b_2^2}= \frac{\partial^2 Q}{\partial b_1\partial b_2}=\sum_{i=1}^nx_i^2$$.

So the determinant of the Hessian is zero.
The function does have a global minimum value ($0$), but the minimizer vector is not unique. This happens because due to the equality constraints, the values of the $X$'s do not affect the solution.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.