0
$\begingroup$

I am having a hard time understanding simple linear regression. I got to a point, in this website to which I can see the closest answer to my question :

To minimize our error function, S, we must find where the first derivative of S is equal to 0 concerning a and b. The closer a and b are to 0, the less total error for each point is. Let’s find the partial derivative of a first... and then they proceed with that calculations.

My logical point which I can not link is, I can understand that the derivative of a quadratic function gives us the slope of the tangent line at that point. I can partially understand why RSS is squared, the only reason I being that is only this mathematical reason of deriving this function afterwards. But I can not understand anything more. All I see is that I have an error function, then I square it only to differentiate it afterwards with respect to minimizers. Why?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Succinctly, we use the squared error in traditional regression, simply because the sum across all $Y_i - \hat{Y_i}=0$ (and the mathematics is a bit easier than using something like the absolute value of the difference). If we square $Y_i - \hat{Y_i}$ and then add these up, across all $i$, the result will not be zero (it would be without the square). Then we want to find where this function is at a minimum, as this would be minimizing the errors. From calculus, this can be found by setting the derivative of the function equal to zero and solving for $a$ and $b$. $\endgroup$
    – StatsStudent
    Commented Sep 11, 2023 at 5:55
  • $\begingroup$ I would suggest you to turn this as an answer @StatsStudent. Perhaps it could be a complement to the existing answer of mine. Of course, if you wish. $\endgroup$
    – User1865345
    Commented Sep 11, 2023 at 6:11
  • $\begingroup$ thank you for your answers. i am still trying to process the new informations before formulating a more intuitive and concise approach of asking what I don't understand $\endgroup$
    – Perja
    Commented Sep 11, 2023 at 18:39
  • 1
    $\begingroup$ Do you get why we take derivatives of the error function, whatever it is, but not get why we choose to square the individual errors? That is a completely legitimate inquiry but quite different from what it appears others have addressed and what I first thought when I saw this. // Related? $\endgroup$
    – Dave
    Commented Sep 11, 2023 at 18:53

1 Answer 1

Reset to default
0
$\begingroup$

Let me sketch a brief picture of the underlying theme of linear regression:

When $\mathbf y=\mathbf X\boldsymbol\beta+\boldsymbol\varepsilon,~\mathbf y\ne \mathbf X\boldsymbol\beta^\star$ for some $\boldsymbol\beta^\star\in\mathbb R^p$ that is, $\mathbf y\notin \mathcal C(\mathbf X). $ This means the system $\mathbf y= \mathbf X\boldsymbol\beta^\star$ is not solvable.

The problem of linear regression ultimately boils down to the more intrinsic question: how to solve a non-solvable linear system?

One can resort to a useful result in functional analysis, known as Closest Point Principle, which states that

Let $U$ be a non-empty closed convex subset of a Hilbert space $\mathcal H;$ let $y\in U^\complement.$ Then there exists a unique $\hat a\in A$ such that $$\Vert y-\hat a\Vert=\inf\{\Vert y-a\Vert:a\in A\}.$$

Simplifying in our perspective, the result assures us that we can find $\hat{\mathbf y}\in\mathcal C(\mathbf X) $ such that it has the closest square distance to $\mathbf y\notin \mathcal C(\mathbf X)$ i.e. $\Vert \mathbf y-\hat{\mathbf y}\Vert^2\leq \Vert\mathbf y-\mathbf y^\star\Vert^2~\forall ~\mathbf y^\star\in \mathcal C(\mathbf X). \tag 1\label 1$ And one can find that $\hat{\mathbf y}=\mathbf P_\mathbf X\mathbf y, $ where $\mathbf P_\mathbf X:=\mathbf X(\mathbf X^\top\mathbf X) ^-\mathbf X^\top$ is the projection operator onto $\mathcal C(\mathbf X)$ and thus $\hat{\mathbf y}=\mathbf X\underbrace{(\mathbf X^\top\mathbf X) ^-\mathbf X^\top\mathbf y}_{:=\hat{\boldsymbol\beta}}.$ When the norm is $\Vert \cdot\Vert_2, ~\eqref 1$ becomes

$$\sum_i (\mathbf y-\mathbf X\hat{\boldsymbol\beta})^2_i\leq \sum_i(\mathbf y-\mathbf X\boldsymbol\beta^\star)^2_i~\forall~\boldsymbol\beta^\star\in\mathbb R^p.\tag 2$$

So far, so good. But what is the working principle? The same old calculus to the rescue:

$$\frac{\partial}{\partial\boldsymbol\beta^\star}\Vert \mathbf y-\mathbf X\boldsymbol\beta^\star\Vert^2=2\mathbf X^\top\mathbf X\boldsymbol\beta^\star-2\mathbf X^\top\mathbf y=\mathbf 0.\tag 3$$

Meanwhile there have been handful of discussions on or around this topic here, which you can look into. One of the more general and expository posts to start with would be this CV post: Why do we usually choose to minimize the sum of square errors (SSE) when fitting a model? and links therein.

References:

$\rm [I]$ Linear Regression, Jürgen Groß, Springer-Varleg, $2003, $ sec. $2.2.1, $ pp. $37-39.$

$\rm [II]$ An Introduction to Functional Analysis, James C. Robinson, Cambridge University Press, $2020, $ sec. $10.1, $ pp. $126-127.$

Improve this answer
$\endgroup$
8
  • 1
    $\begingroup$ While this is a correct answer, I'm guessing an answer involving Hilbert spaces and your notation will not prove to be very useful to OP since it seems she is struggling with basic concepts in regression. If the OP isn't quite grasping these concepts, I'm guessing an explanation involving functional analysis, Hilbert spaces, and more complex notation will not be useful. I'd suggest modifying your response to be more aligned with the experience/educational level of the OP. $\endgroup$
    – StatsStudent
    Commented Sep 11, 2023 at 5:53
  • $\begingroup$ Thanks for the feedback. The principle has been stated for completion. How that works from the pov of OP has also been stated subsequently. If one wants to present the fundamental underlying principle, one can not shed all the mathematical jargons, imo. Still if you wish, I would water down the already what I deem the simplest way I could have provided a crux of. $\endgroup$
    – User1865345
    Commented Sep 11, 2023 at 6:06
  • 1
    $\begingroup$ As for complex notation, I have to say I have always embraced usage of the conventional notations and symbols most books have adapted into. $\endgroup$
    – User1865345
    Commented Sep 11, 2023 at 6:08
  • $\begingroup$ I probably didn't convey that well -- I didn't mean to imply that the notation isn't conventional. I simply meant to convey that the notation is probably difficult for the OP to understand given she seems to be struggling with some basic concepts in mathematics/statistics/calculus. I'm assuming that she's likely not to benefit greatly from what the OP would probably consider a more heavy notational explanation and a rigorous mathematical explanation. Instead, I'd argue that a plain English explanation might be more suited to allow the OP to comprehend what's really going on... (con't) $\endgroup$
    – StatsStudent
    Commented Sep 11, 2023 at 6:18
  • 1
    $\begingroup$ "under the hood." But that's just my perspective, and may not be that of others. Of course, feel free to do as you wish, but I do think a brief plain English explanation would go a long way for the OP, and would be more likely to earn you a checkmark. $\endgroup$
    – StatsStudent
    Commented Sep 11, 2023 at 6:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.