5
$\begingroup$

What is the linear algebraic and geometric interpretation of the sentence found on this post:

When $y = X\beta + e$, the least squares problem which imposes a spherical restriction $\delta$ on the value of $\beta$, [which] can be written as

\begin{equation} \begin{array} &\operatorname{min}\ \| y - X\beta \|^2_2 \\ \operatorname{s.t.}\ \ \|\beta\|^2_2 \le \delta^2 \end{array} \end{equation}

It sounds as though it has to be connected with the concept of spherical errors, but there is likely more to it. Further I was intrigued by this illustration comparing OLS to ridge regression:

enter image description here

Source: A First Course in Linear Model Theory / Edition 1 by Nalini Ravishanker, Dipak K. Dey, Dipak K. Dey

with the text:

We can characterize ridge regression as a restricted least squares problem. Consider the least squares in the centered and scaled multiple regression model $\bf y^*=X^*\beta^*+\varepsilon$ subject to the spherical restriction $$\beta^{*'}\beta^*\leq d^2$$ for a given value $d^2.$

Continuing along the hints given by W. Huber in the comments, I wonder if this is related to the ellipsoid representation, and the geometric interpretation of ridge regression as a contained OLS optimized at the locus of osculation determined by $\mathbf \beta^\top K \beta$:

enter image description here

Source: Elliptical Insights: Understanding Statistical Methods through Elliptical Geometry Statistical Science 28(1) · February 2013, Michael Friendly and Georges Monette

Improve this question
$\endgroup$
10
  • 3
    $\begingroup$ For $\beta\in\mathbb{R}^k$, the equation $||\beta||^2\le\delta^2$ is satisfied by the points in the ball (aka "sphere") of radius $\delta$ centered at the origin of $\mathbb{R}^k$. I believe that's all that the phrase "spherical restriction" was intended to mean. $\endgroup$
    – whuber ♦
    Commented Jan 18, 2017 at 16:51
  • $\begingroup$ @whuber Even centered, though, each $ \beta_i$ coefficient is different. Do you draw this hypersphere with a radius equal to the smallest coefficient? And is it just a statement implying that the coefficients are all less than infinity? I am still not seeing it... $\endgroup$
    – Antoni Parellada
    Commented Jan 18, 2017 at 16:56
  • $\begingroup$ You seem to be confusing a ball with a line. The constraints require only that $\beta$ lie within the ball, not that it lie on the line $\beta_1=\beta_2=\cdots=\beta_k$! Note that both $\delta$ and $d$ are "given values": that is, they are not free to vary, but are specified as part of the problem. $\endgroup$
    – whuber ♦
    Commented Jan 18, 2017 at 16:58
  • $\begingroup$ @whuber I see the picture now - not on the surface of the sphere, but within. Where are the values of $\delta$ specified? $\endgroup$
    – Antoni Parellada
    Commented Jan 18, 2017 at 16:59
  • 1
    $\begingroup$ After Ridge Regression was invented, it was re-interpreted as the solution to OLS with a particular kind of Bayes prior on $\beta$. Within that interpretation, $\delta$ does have some meaning (related to how small you think the coefficients ought to be). $\endgroup$
    – whuber ♦
    Commented Jan 18, 2017 at 17:07

1 Answer 1

Reset to default
2
$\begingroup$

As explained in the comments, the problem is ridge regression, where the squared error is minimized subject to a bound on the $\ell_2$ norm of $\beta$.

As far as I can discern, constraining the $\ell_2$ norm of $\beta$ is not connected to linear model assumptions that the error is spherical. After all, using Bayesian language, the likelihood $\pi(y|\beta)$ being spherical does not suggest that the prior $\pi(\beta)$ should be too.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.