Return to Question

What is the linear algebraic and geometric interpretation of the sentence found on this post:

When $y = X\beta + e$, the least squares problem which imposes a spherical restriction $\delta$ on the value of $\beta$, [which] can be written as

\begin{equation} \begin{array} &\operatorname{min}\ \| y - X\beta \|^2_2 \\ \operatorname{s.t.}\ \ \|\beta\|^2_2 \le \delta^2 \end{array} \end{equation}

It sounds as though it has to be connected with the concept of spherical errors, but there is likely more to it. Further I was intrigued by this illustration comparing OLS to ridge regression:

Source: A First Course in Linear Model Theory / Edition 1 by Nalini Ravishanker, Dipak K. Dey, Dipak K. Dey

with the text:

We can characterize ridge regression as a restricted least squares problem. Consider the least squares in the centered and scaled multiple regression model $\bf y^*=X^*\beta^*+\varepsilon$ subject to the spherical restriction $$\beta^{*'}\beta^*\leq d^2$$ for a given value $d^2.$

Continuing along the hints given by W. Huber in the comments, I wonder if this is related to the ellipsoid representation, and the geometric interpretation of ridge regression as a contained OLS optimized at the locus of osculation determined by $\mathbf \beta^\top K \beta$:

Source: Elliptical Insights: Understanding Statistical Methods through Elliptical Geometry Statistical Science 28(1) · February 2013, Michael Friendly and Georges Monette

What is the linear algebraic and geometric interpretation of the sentence found on this post:

When $y = X\beta + e$, the least squares problem which imposes a spherical restriction $\delta$ on the value of $\beta$, [which] can be written as

\begin{equation} \begin{array} &\operatorname{min}\ \| y - X\beta \|^2_2 \\ \operatorname{s.t.}\ \ \|\beta\|^2_2 \le \delta^2 \end{array} \end{equation}

It sounds as though it has to be connected with the concept of spherical errors, but there is likely more to it. Further I was intrigued by this illustration comparing OLS to ridge regression:

Source: A First Course in Linear Model Theory / Edition 1 by Nalini Ravishanker, Dipak K. Dey, Dipak K. Dey

with the text:

We can characterize ridge regression as a restricted least squares problem. Consider the least squares in the centered and scaled multiple regression model $\bf y^*=X^*\beta^*+\varepsilon$ subject to the spherical restriction $$\beta^{*'}\beta^*\leq d^2$$ for a given value $d^2.$

Continuing along the hints given by W. Huber in the comments, I wonder if this is related to the ellipsoid representation, and the geometric interpretation of ridge regression as a contained OLS optimized at the locus of osculation determined by $\mathbf \beta^\top K \beta$:

Source: Elliptical Insights: Understanding Statistical Methods through Elliptical Geometry Statistical Science 28(1) · February 2013, Michael Friendly and Georges Monette

What is the linear algebraic and geometric interpretation of the sentence found on this post:

When $y = X\beta + e$, the least squares problem which imposes a spherical restriction $\delta$ on the value of $\beta$, [which] can be written as

\begin{equation} \begin{array} &\operatorname{min}\ \| y - X\beta \|^2_2 \\ \operatorname{s.t.}\ \ \|\beta\|^2_2 \le \delta^2 \end{array} \end{equation}

It sounds as though it has to be connected with the concept of spherical errors, but there is likely more to it. Further I was intrigued by this illustration comparing OLS to ridge regression:

Source: A First Course in Linear Model Theory / Edition 1 by Nalini Ravishanker, Dipak K. Dey, Dipak K. Dey

with the text:

We can characterize ridge regression as a restricted least squares problem. Consider the least squares in the centered and scaled multiple regression model $\bf y^*=X^*\beta^*+\varepsilon$ subject to the spherical restriction $$\beta^{*'}\beta^*\leq d^2$$ for a given value $d^2.$

What is the linear algebraic and geometric interpretation of the sentence found on this post:

When $y = X\beta + e$, the least squares problem which imposes a spherical restriction $\delta$ on the value of $\beta$, [which] can be written as

\begin{equation} \begin{array} &\operatorname{min}\ \| y - X\beta \|^2_2 \\ \operatorname{s.t.}\ \ \|\beta\|^2_2 \le \delta^2 \end{array} \end{equation}

It sounds as though it has to be connected with the concept of spherical errors, but there is likely more to it. Further I was intrigued by this illustration comparing OLS to ridge regression:

Source: A First Course in Linear Model Theory / Edition 1 by Nalini Ravishanker, Dipak K. Dey, Dipak K. Dey

with the text:

We can characterize ridge regression as a restricted least squares problem. Consider the least squares in the centered and scaled multiple regression model $\bf y^*=X^*\beta^*+\varepsilon$ subject to the spherical restriction $$\beta^{*'}\beta^*\leq d^2$$ for a given value $d^2.$

Continuing along the hints given by W. Huber in the comments, I wonder if this is related to the ellipsoid representation, and the geometric interpretation of ridge regression as a contained OLS optimized at the locus of osculation determined by $\mathbf \beta^\top K \beta$:

Source: Elliptical Insights: Understanding Statistical Methods through Elliptical Geometry Statistical Science 28(1) · February 2013, Michael Friendly and Georges Monette

What is the linear algebraic and geometric interpretation of the sentence found on this post:

When $y = X\beta + e$, the least squares problem which imposes a spherical restriction $\delta$ on the value of $\beta$, [which] can be written as

\begin{equation} \begin{array} &\operatorname{min}\ \| y - X\beta \|^2_2 \\ \operatorname{s.t.}\ \ \|\beta\|^2_2 \le \delta^2 \end{array} \end{equation}

It sounds as though it has to be connected with the concept of spherical errors, but there is likely more to it. Further I was intrigued by this illustration comparing OLS to ridge regression:

with the text:

We can characterize ridge regression as a restricted least squares problem. Consider the least squares in the centered and scaled multiple regression model $\bf y^*=X^*\beta^*+\varepsilon$ subject to the spherical restriction $$\beta^{*'}\beta^*\leq d^2$$ for a given value $d^2.$

What is the linear algebraic and geometric interpretation of the sentence found on this post:

When $y = X\beta + e$, the least squares problem which imposes a spherical restriction $\delta$ on the value of $\beta$, [which] can be written as

\begin{equation} \begin{array} &\operatorname{min}\ \| y - X\beta \|^2_2 \\ \operatorname{s.t.}\ \ \|\beta\|^2_2 \le \delta^2 \end{array} \end{equation}

It sounds as though it has to be connected with the concept of spherical errors, but there is likely more to it. Further I was intrigued by this illustration comparing OLS to ridge regression:

Source: A First Course in Linear Model Theory / Edition 1 by Nalini Ravishanker, Dipak K. Dey, Dipak K. Dey

with the text:

We can characterize ridge regression as a restricted least squares problem. Consider the least squares in the centered and scaled multiple regression model $\bf y^*=X^*\beta^*+\varepsilon$ subject to the spherical restriction $$\beta^{*'}\beta^*\leq d^2$$ for a given value $d^2.$