8
$\begingroup$

For the linear regression RSS:

$$ RSS = \sum_{i=1}^{n}\left(Y_i-\sum_{j=1}^{p} X_{ij}\beta_j\right)^2 $$

by decomposing it we have something like

$$ \beta_1^2X_{ij}^2 - \beta_2^2X_{ij}^2 - \beta_1X_{ij}Y_{i} - \beta_2X_{ij}Y_{i} + .... $$

so that the terms are squared in the beta's. I have often seen plots of $\beta_1$ on the x-axis and $\beta_2$ on the y-axis being contours like

enter image description here

How exactly are these red circles constructed? Thank you

$\endgroup$

1 Answer 1

11
$\begingroup$

Say we are in a simple linear regression setting, so $\beta = \left(\beta_1,\beta_2\right)$ and $Y_i = \beta_1 + X_i\beta_2 + \epsilon_i$ where $\epsilon_i ~ \sim N\left(0,1\right)$ then \begin{align} RSS & = \sum_{i=1}^{n}\left(Y_i - \beta_1 - X_i\beta_2 \right)^2\\ & =\sum_{i=1}^{n}\left( Y_i^2 - Y_i\beta_1 -Y_i X_i \beta_2 + 2\beta_1 X_i \beta_2 + \beta_1^2 + X_i^2\beta_2^2\right) \\ & = \sum_{i=1}^{n} Y_i^2 - \beta_1 \sum_{i=1}^{n} Y_i - \beta_2 \sum_{i=1}^{n} Y_i X_i + 2\beta_1\beta_2 \sum_{i=1}^{n} X_i + n\beta_1^2 + \beta_2^2 \sum_{i=1}^{n} X_i^2 \end{align}

Which is of the form

\begin{align} A\beta_1^2 + B\beta_1\beta_2 + C\beta_2^2 + D\beta_1 + E\beta_2 + F = 0 \end{align}

for a fixed value of RSS. This is the equation of a conic https://en.wikipedia.org/wiki/Conic_section and we can find what type by considering the discriminant

\begin{align} B^2 - 4AC & = \left(2\sum_{i=1}^{n}X_i\right)^2 - 4n\sum_{i=1}^n X_i^2 \\ & = -4 \left[ \left(\sum_{i=1}^n X_i\right)^2 - n \sum_{i=1}^n X_i^2 \right] \\ & = -4 \left[ \sum_{i = 1}^n \left( X_i - \sum_{i = 1}^n X_i \right)^2 \right] \leq 0 \end{align}

So assuming the $X_i$ are not all equal, the discriminant is strictly less than 0, and the equation gives an ellipse.

$\endgroup$
5
  • $\begingroup$ No need to check the determinant: the very definition of the RSS as a sum of squares immediately shows it's an ellipse. $\endgroup$
    – whuber
    Commented Apr 28, 2021 at 16:43
  • $\begingroup$ Does the same hold also for the log-likelihood loss function? Are its contous ellipses? $\endgroup$ Commented Feb 12, 2023 at 19:21
  • $\begingroup$ @adosar if you're doing linear regression with independent normal errors then I believe so - if you write it out the log-likelihood should be a linear transformation of the RSS and so also an ellipse. $\endgroup$ Commented Feb 16, 2023 at 16:43
  • $\begingroup$ @whuber Can you elaborate why we don't need to check the discriminant? $\endgroup$ Commented Mar 25, 2023 at 11:06
  • $\begingroup$ @adosar A sum of squares is never negative. Ergo, the conic can only be an ellipse, parabola, or pair of lines; and when it is nondegenerate (as illustrated), it must be an ellipse. $\endgroup$
    – whuber
    Commented Mar 25, 2023 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.