1
$\begingroup$

I was trying to understand geometric interpretation of regularization and came across following statement here:

$$\text{Mean Square Error}\; E(y,\hat{y})=\frac{1}{n}\lVert\hat{y}-y\rVert^2$$ $$=\frac{1}{n}(b^TX^TXb-2b^TX^Ty+y^Ty)$$ Since $X^TX$ is positive semidefinite, we know that $b^TX^TXb\geq0$. Furthermore, we know that (from vector calculus) it will be a paraboloid (bowl-shaped surface) in the $(E,b_1,b_2)$ space. The following diagram depicts this situation. enter image description here

I am not able to get the sentence above. Specifically I didnt get "positive semidefinite" and "we know from vector calculas that it will be a paraboloid". How / why the mean square error surface is bowl shaped? Is their intuition (visual or geometrical if possible) behind it?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ The definition is explicitly proportional to a sum of squares: that's what $||\hat y - y||^2$ means. The graphs of sums of squares are paraboloids. Try it with one or two variables where you can plot the graph and see it. "Bowl-shaped surface" is not quite correct, however: the "bowl" can be like a curved sheet of paper (similar to the first plot in my post at stats.stackexchange.com/a/7629/919) and it need not be circularly symmetric. $\endgroup$
    – whuber ♦
    Commented Nov 30, 2022 at 20:32
  • $\begingroup$ Lots of helpful information in stats.stackexchange.com/questions/224005/… $\endgroup$
    – Sycorax ♦
    Commented Nov 30, 2022 at 20:32
  • $\begingroup$ @whuber is it just that $x^2$ in 2D is parabola. So, in 3D, its paraboloids? $\endgroup$
    – Mahesha999
    Commented Nov 30, 2022 at 20:56
  • $\begingroup$ Yes. There are a limited number of basic shapes of the graphs of quadratic forms. Linear algebra shows how to find the shape by diagonalizing the form: in $n$ dimensions there are coordinates $(x_1,x_2,\ldots,x_n)$ in which the shape is the graph of $x_1^2+\cdots+x_r^2-(x_{r+1}^2+\cdots+x_{r+s}^2)$ where $0\le r+s\le n.$ The numbers $r,s,n$ determine the shape, which is called a "paraboloid" when either $r=0$ or $s=0.$ That this shape is unique is called Sylvester's Law of Inertia. $\endgroup$
    – whuber ♦
    Commented Nov 30, 2022 at 21:15

1 Answer 1

Reset to default
1
$\begingroup$

Suppose $b$ is 1 dimensional. Then you have a loss function of the shape

$$ a_1 \cdot b^2 + a_2 \cdot b + a_3 $$

Which is a parabola. If $a_1 > 0$ then it will be a "smiling" / "convex" / "upward facing" parabola.

This generalizes to bigger dimensions:

$$b^T A_1b + b^T a_2 + a_3 $$

Only now, for it to be "smiling" we require $A_1 \succ 0$ to be positive-definite.

Improve this answer
$\endgroup$
1
  • $\begingroup$ The graphs of non-definite forms are hyperboloids. $\endgroup$
    – whuber ♦
    Commented Dec 1, 2022 at 14:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.