1
$\begingroup$

I'm a B.S. Math graduate who likes to (attempt) to teach myself statistics on my own time because I can't afford a masters degree.

It really bothers me when I can't understand at a fundamental level how and why something works and so in this post I wanted to provide every single step in how coefficients are derived using the least squares method in linear regression. Most people here probably know this sort of thing a lot better than I do but it was satisfying to see it all come together and I hope it is for you too.

This covers the multivariable (nth dimensional) case as the 1 variable case is just a simpler version of it. At the end there is a simple example in R showing that the result obtained from scratch is correct.

$\endgroup$
0

1 Answer 1

1
$\begingroup$

The linear regression model is given by $\hat{y}_i$ = $\hat{\beta_0} + \hat{\beta_1}x_{i, 1} + ... + \hat{\beta_n}x_{i, n}$ for observations $i = 1, 2, ..., n$, or $n$ observations total. The squared error between the predicted ($\hat{y}_i$) and observed ($y_i$) is given by $(y_i - \hat{y}_i)^2$ for a single observation and over $n$ observations it is given by $\sum_{i = 1}^{n} (y_i - \hat{y}_i)^2$ where, writing it out, the sum is $((y_1 - \hat{y}_1)^2 + (y_2 - \hat{y}_2)^2 + ... + (y_n - \hat{y}_n)^2)$.

Since $\hat{y}_i = \hat{\beta_0} + \hat{\beta_1}x_{i, 1} + ... + \hat{\beta_n}x_{i, n}$ we can substitute into the above expression and obtain $((y_1 - \hat{\beta_0} + \hat{\beta_1}x_{1, 1} + ... + \hat{\beta_n}x_{1, n})^2 + (y_2 - \hat{\beta_0} + \hat{\beta_1}x_{2, 1} + ... + \hat{\beta_n}x_{2, n})^2 + ... + (y_n - \hat{\beta_0} + \hat{\beta_1}x_{n, 1} + ... + \hat{\beta_n}x_{n, n})$ which is equal to the expression usually seen in textbooks, $\sum_{i = 1}^{n} (y_i - \hat{\beta_0} + \hat{\beta_1}x_{i, 1} + ... + \hat{\beta_n}x_{i, n})^2$, the $SSE$.

The sum of squared error can be rewritten as a matrix vector product. Let

$\vec{\hat{\beta}} = \begin{bmatrix} \hat{\beta}_0 \\ \hat{\beta}_1 \\ ... \\ \hat{\beta}_n \end{bmatrix}$ be the $n + 1$ x $1$ vector of estimated coefficients

$\vec{\hat{y}} = \begin{bmatrix} \hat{y}_1 \\ \hat{y}_2 \\ ... \\ \hat{y}_n \end{bmatrix}$ be the $n$ x $1$ vector of predicted $y$ values

and let $X = \begin{bmatrix} 1 & x_{1, 1} & ... & x_{1, n} \\ 1 & x_{2, 1} & ... & x_{2, n} \\ ... & ... & ... & ... \\ 1 & x_{n, 1} & ... & x_{n, n} \end{bmatrix}$ be the $n$ x $n + 1$ design matrix

Then the matrix vector product $X\vec{\hat{\beta}}$ is a product between a $n$ x $n + 1$ matrix and a $n x 1$ by $1$ vector, resulting in a $n$ x $1$ vector

$\begin{bmatrix} 1 & x_{1, 1} & ... & x_{1, n} \\ 1 & x_{2, 1} & ... & x_{2, n} \\ ... & ... & ... & ... \\ 1 & x_{n, 1} & ... & x_{n, n} \end{bmatrix} \begin{bmatrix} \hat{\beta}_0 \\ \hat{\beta}_1 \\ ... \\ \hat{\beta}_n \end{bmatrix} = \begin{bmatrix} \hat{\beta}_0*1 + \hat{\beta}_1*x_{1, 1} + ... + \hat{\beta}_n * x_{1, n} \\ \hat{\beta}_0 * 1 + \hat{\beta}_1 * x_{2, 1} + ... + \hat{\beta}_n * x_{2, n} \\ ... \\ \hat{\beta}_0 * 1 + \hat{\beta}_1 * x_{n, 1} + ... + \hat{\beta}_n * x_{1, n} \end{bmatrix}$

where $\hat{\beta}_0*1 + \hat{\beta}_1*x_{1, 1} + ... + \hat{\beta}_n * x_{1, n} = \hat{y}_1$

$\hat{\beta}_0 * 1 + \hat{\beta}_1 * x_{2, 1} + ... + \hat{\beta}_n * x_{2, n} = \hat{y}_2$

and $\hat{\beta}_0 * 1 + \hat{\beta}_1 * x_{n, 1} + ... + \hat{\beta}_n * x_{1, n} = \hat{y}_n$

thus the vector resulting from the matrix vector product is equal to $\vec{\hat{y}}$, or the predicted $y$ values. Then this is a matrix vector expression for the linear model. Since we can also write the observed $y$ values as a vector then we can write the difference between every $y_i$ and $\hat{y}_i$ as

$\begin{bmatrix} y_1 \\ y_2 \\ ... \\ y_n \end{bmatrix}$ - $\begin{bmatrix} 1 & x_{1, 1} & ... & x_{1, n} \\ 1 & x_{2, 1} & ... & x_{2, n} \\ ... & ... & ... & ... \\ 1 & x_{n, 1} & ... & x_{n, n} \end{bmatrix} \begin{bmatrix} \hat{\beta}_0 \\ \hat{\beta}_1 \\ ... \\ \hat{\beta}_n \end{bmatrix} = \begin{bmatrix} e_1 \\ e_2 \\ ... \\ e_n \end{bmatrix}$

Since this represents the difference, or error, between every actual and predicted response value, and it is a vector then squaring it should give the $SSE$. That is due to us taking a dot product between the error vector and its transpose which will give us the sum of every squared error term, or the sum of squared errors

$\begin{bmatrix} e_1 \\ e_2 \\ ... \\ e_n \end{bmatrix} \begin{bmatrix} e_1 \\ e_2 \\ ... \\ e_n \end{bmatrix} = \begin{bmatrix} e_1 \\ e_2 \\ ... \\ e_n \end{bmatrix}^T \begin{bmatrix} e_1 \\ e_2 \\ ... \\ e_n \end{bmatrix} = (e_1)^2 + (e_2)^2 + ... + (e_n)^2 = \sum_{i = 1}^{n} (e_i)^2$

which it does. Thus we can write the $SSE$ in terms of $\vec{\hat{\beta}}$ as ($\vec{y} - X\vec{\hat{\beta}})^T(\vec{y} - X\vec{\hat{\beta}})$. To rewrite this expression we recall that $X\vec{\hat{\beta}}$ is a $n$ x $1$ vector thus its transpose should be a $1$ x $n$ vector and this is only possible when we have a $1$ x $n + 1$ vector and a $n + 1$ x $n$ matrix multiplied in the direction $\vec{\hat{\beta}}^T X^T$. This gives us ($\vec{y}^T - \vec{\hat{\beta}}^T X^T)(\vec{y} - X\vec{\hat{\beta}})$ which becomes $\vec{y}^T\vec{y} - \vec{y}^T X\vec{\hat{\beta}} - \vec{\hat{\beta}}^T X^T \vec{y} + \vec{\hat{\beta}}^T X^TX\vec{\hat{\beta}}$ by distributing the terms.

What we want to do in least squares is find the vector of coefficients that minimize the function (the sum of squared errors), which we can do by taking the derivative and setting it equal to $0$. The $SSE$ is useful for this purpose as it is a convex function, meaning its shaped akin to a parabola like $y = x^2$, and must have a single global minimum. Thus when we find the derivative we've found the coefficients that minimize the $SSE$.

The derivative with respect to $\vec{\hat{\beta}}$ is given by $\frac{\partial}{\partial \vec{\hat{\beta}}} (\vec{y}^T\vec{y} - \vec{y}^T X\vec{\hat{\beta}} - \vec{\hat{\beta}}^T X^T \vec{y} + \vec{\hat{\beta}}^T X^TX\vec{\hat{\beta}})$. $\frac{\partial}{\partial \vec{\hat{\beta}}} (\vec{y}^T\vec{y}) = 0$ as there are no $\hat{\beta}$ terms. The derivatives of the other two terms will be shown in detail.

Suppose that we have some vector $\vec{x}$ and matrix $A$ multiplied in the form $\vec{x}^TA\vec{x}$ where

$\vec{x} = \begin{bmatrix} x_1 \\ ... \\ x_n \end{bmatrix}$ and $A = \begin{bmatrix} a_{1,1} & ... & a_{1, n} \\ ... & ... & ... \\ a_{n, 1} & ... & a_{n, n} \end{bmatrix}$ where $\vec{x}$ is a $n$ x $1$ vector and $A$ is a $n$ x $n$ matrix

Then $\vec{x}^T A\vec{x} = \begin{bmatrix} x_1 \\ x_2 \\ ... \\ x_n \end{bmatrix}^T \begin{bmatrix} a_{1,1} & ... & a_{1, n} \\ a_{2, 1} & ... & a_{2, n} \\ ... & ... & ... \\ a_{n, 1} & ... & a_{n, n} \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ ... \\ x_n \end{bmatrix}$ the first product is equal to a $1$ x $n$ vector of the form

$\vec{x}^TA = \begin{bmatrix} x_1a_{1,1} + x_2a_{2, 1} + ... + x_na_{n, 1} \\ ... \\ x_1a_{1, n} + x_2a_{2, n} + ... + x_na_{n, n} \end{bmatrix}^T$ and multiplying this to the $n$ x $1$ vector $\vec{x}$ returns

$\begin{bmatrix} x_1a_{1,1} + x_2a_{2, 1} + ... + x_na_{n, 1} \\ ... \\ x_1a_{1, n} + x_2a_{2, n} + ... + x_na_{n, n} \end{bmatrix}^T \begin{bmatrix} x_1 \\ x_2 \\ ... \\ x_n \end{bmatrix} = x_1(x_1a_{1,1} + x_2a_{2, 1} + ... + x_na_{n, 1}) + x_2(x_1a_{1, 2} + x_2a_{2, 2} + ... + x_na_{n, 2}) + ... + x_n(x_1a_{1, n} + x_2a_{2, n} + ... + x_na_{n, n})$. The derivative of this sum with respect to $\vec{x}$ is given by

$\frac{\partial \vec{x}^TA\vec{x}}{\partial \vec{x}} = \begin{bmatrix} \frac{\partial \vec{x}^TA\vec{x}}{\partial x_1} \\ \frac{\partial \vec{x}^TA\vec{x}}{\partial x_2} \\ ... \\ \frac{\partial \vec{x}^TA\vec{x}}{\partial x_n} \end{bmatrix}$. Suppose we take the partial with respect to $x_1$. Then we have $\frac{\partial}{\partial x_1} (x_1(x_1a_{1,1} + x_2a_{2, 1} + ... + x_na_{n, 1}) + x_2(x_1a_{1, 2} + x_2a_{2, 2} + ... + x_na_{n, 2}) + ... + x_n(x_1a_{1, n} + x_2a_{2, n} + ... + x_na_{n, n})) = (x_1a_{1,1} + x_2a_{2, 1} + ... + x_na_{n, 1}) + x_1a_{1,1} + x_2a_{1, 2} + ... + x_na_{1, n}$.

Similarly,

$\frac{\partial}{\partial x_n} (x_1(x_1a_{1,1} + x_2a_{2, 1} + ... + x_na_{n, 1}) + x_2(x_1a_{1, 2} + x_2a_{2, 2} + ... + x_na_{n, 2}) + ... + x_n(x_1a_{1, n} + x_2a_{2, n} + ... + x_na_{n, n})) = x_1a_{n, 1} + x_2a_{n, 2} + ... + (x_1a_{1, n} + x_2a_{2, n} + ... + x_na_{n, n}) + x_na_{n,n}$

This is equal to the $n$th row of $A +$ the transpose of the $n$th column of $A$ all multiplied by $\vec{x}$. The reason why the columns of $A$ are transposed in the operation is because, in vector form, the columns $a_{1,1}, ..., a_{1, n}$ form a 1 x $n$ vector while the rows $a_{1,1}, ..., a_{n, 1} are a $n$ x $1$ vector and to add the two vectors together (in order to obtain the observed result in the derivative) one must be transposed. The result, already described in words is

$\frac{\partial \vec{x}^TA\vec{x}}{\partial \vec{x}} = (A + A^T)\vec{x}$

and when $A$ is symmetric - is equal to its transpose - then this becomes $(A + A)\vec{x} = 2A\vec{x}$

Returning to the original problem, we have $\frac{\partial}{\partial \vec{\hat{\beta}}} (\vec{y}^T\vec{y} - \vec{y}^T X\vec{\hat{\beta}} - \vec{\hat{\beta}}^T X^T \vec{y} + \vec{\hat{\beta}}^T X^TX\vec{\hat{\beta}})$. We can think about $X^TX$ being one big matrix $A$ which is both square and symmetric.

$X = \begin{bmatrix} 1 & x_{1, 1} & ... & x_{1, n} \\ 1 & x_{2, 1} & ... & x_{2, n} \\ ... & ... & ... & ... \\ 1 & x_{n, 1} & ... & x_{n, n} \end{bmatrix}$ which is a $n$ x $n + 1$ matrix

$X^T = \begin{bmatrix} 1 & 1 & ... & 1 \\ x_{1, 1} & x_{2, 1} & ... & x_{n, 1} \\ ... & ... & ... & ... \\ x_{1, n} & x_{2, n} & ... & x_{n, n} \end{bmatrix}$ which is a $n + 1$ x $n$ matrix

$A = X^TX = \begin{bmatrix} 1 & 1 & ... & 1 \\ x_{1, 1} & x_{2, 1} & ... & x_{n, 1} \\ ... & ... & ... & ... \\ x_{1, n} & x_{2, n} & ... & x_{n, n} \end{bmatrix} \begin{bmatrix} 1 & x_{1, 1} & ... & x_{1, n} \\ 1 & x_{2, 1} & ... & x_{2, n} \\ ... & ... & ... & ... \\ 1 & x_{n, 1} & ... & x_{n, n} \end{bmatrix}$ =

$\begin{bmatrix} (1 * 1 + 1 * 1 + ... + 1 * 1) & (1 * x_{1, 1} + 1 * x_{2, 1} + ... + 1 * x_{n, 1}) & ... & (1 * x_{1, n} + 1 * x_{2, n} + ... + 1 * x_{n, n}) \\ (x_{1, 1} * 1 + x_{2, 1} * 1 + ... + x_{n, 1} * 1) & (x_{1, 1} * x_{1, 1} + x_{2, 1} * x_{2, 1} + ... + x_{n, 1} * x_{n, 1}) & ... & (x_{1, 1} * x_{1, n} + x_{2, 1} * x_{2, n} + ... + x_{n, 1} * x_{n, n}) \\ ... & ... & ... & ... \\ (x_{1, n} * 1 + x_{2, n} * 1 + ... + x_{n, n} * 1) & (x_{1, n} * x_{1, 1} + x_{2, n} * x_{2, 1} + ... + x_{n, n} * x_{n, 1}) & ... & (x_{1, n} * x_{1, n} + x_{2, n} * x_{2, n} + ... + x_{n, n} * x_{n, n}) \end{bmatrix}$

which is an $n$ x $n$ matrix. Expanding it out a bit more should allow us to see that it is symmetric

$\begin{bmatrix} (1 * 1 + 1 * 1 + ... + 1 * 1) & (1 * x_{1, 1} + 1 * x_{2, 1} + ... + 1 * x_{n, 1}) & (1 * x_{1, 2} + 1 * x_{2, 2} + ... + 1 * x_{n, 2}) & ... & (1 * x_{1, n} + 1 * x_{2, n} + ... + 1 * x_{n, n}) \\ (x_{1, 1} * 1 + x_{2, 1} * 1 + ... + x_{n, 1} * 1) & (x_{1, 1} * x_{1, 1} + x_{2, 1} * x_{2, 1} + ... + x_{n, 1} * x_{n, 1}) & (x_{1, 1} * x_{1, 2} + x_{2, 1} * x_{2, 2} + ... + x_{n, 1} * x_{n, 2}) & ... & (x_{1, 1} * x_{1, n} + x_{2, 1} * x_{2, n} + ... + x_{n, 1} * x_{n, n}) \\ (x_{1, 2} * 1 + x_{2, 2} * 1 + ... + x_{n, 2} * 1) & (x_{1, 2} * x_{1, 1} + x_{2, 2} * x_{2, 1} + ... + x_{n, 2} * x_{n, 1}) & (x_{1, 2} * x_{1, 2} + x_{2, 2} * x_{2, 2} + ... + x_{n, 2} * x_{n, 2} & ... & (x_{1, 2} * x_{1, n} + x_{2, 2} * x_{2, n} + ... + x_{n, 2} * x_{n, n}) \\ ... & ... & ... & ... & ... \\ (x_{1, n} * 1 + x_{2, n} * 1 + ... + x_{n, n} * 1) & (x_{1, n} * x_{1, 1} + x_{2, n} * x_{2, 1} + ... + x_{n, n} * x_{n, 1}) & (x_{1, n} * x_{1, 2} + x{2, n} * x_{2, 2} + ... + x_{n, n} * x_{n, 2}) & ... & (x_{1, n} * x_{1, n} + x_{2, n} * x_{2, n} + ... + x_{n, n} * x_{n, n}) \end{bmatrix}$

thus we can rewrite $\vec{\hat{\beta}}^T X^TX\vec{\hat{\beta}}$ as $\vec{\hat{\beta}}^TA\vec{\hat{\beta}}$ and since $A$ is symmetric then $\frac{\partial}{\partial \vec{\hat{\beta}}} \vec{\hat{\beta}}^TA\vec{\hat{\beta}} = 2A\vec{\hat{\beta}} = 2X^TX\vec{\hat{\beta}}$

The last part of obtaining the derivative involves determining $\frac{\partial}{\partial \vec{\hat{\beta}}} (- \vec{y}^T X\vec{\hat{\beta}} - \vec{\hat{\beta}}^T X^T \vec{y})$

To see how this is calculated it is best to write it all out beginning with $- \vec{y}^T X\vec{\hat{\beta}}$

$- \vec{y}^T X\vec{\hat{\beta}} = -(\begin{bmatrix} y_1 \\ y_2 \\ ... \\ y_n \end{bmatrix}^T \begin{bmatrix} 1 & x_{1, 1} & ... & x_{1, n} \\ 1 & x_{2, 1} & ... & x_{2, n} \\ ... & ... & ... & ... \\ 1 & x_{n, 1} & ... & x_{n, n} \end{bmatrix} \begin{bmatrix} \hat{\beta}_0 \\ \hat{\beta}_1 \\ ... \\ \hat{\beta}_n \end{bmatrix}) = -(\begin{bmatrix} (y_1 + y_2 + ... y_n) \\ (y_1 * x_{1, 1} + y_2 * x_{2, 1} + ... + y_n * x_{n, 1}) \\ ... \\ (y_1 * x_{1, n} + y_2 * x_{2, n} + ... + y_n * x_{n, n}) \end{bmatrix}^T \begin{bmatrix} \hat{\beta}_0 \\ \hat{\beta}_1 \\ ... \\ \hat{\beta}_n \end{bmatrix})$

where we have a $1$ x $n + 1$ vector multiplied to a $n + 1$ x $1$ vector, thus this is another dot product that will result in a big sum given by

$-(\hat{\beta}_0(y_1 + y_2 + ... y_n) + \hat{\beta}_1(y_1 * x_{1, 1} + y_2 * x_{2, 1} + ... + y_n * x_{n, 1}) + ... + \hat{\beta}_n(y_1 * x_{1, n} + y_2 * x_{2, n} + ... + y_n * x_{n, n})$

and its derivative is much easier to compute now that we have it in this form.

$\frac{\partial}{\partial \vec{\hat{\beta}}} (-(\hat{\beta}_0(y_1 + y_2 + ... y_n) + \hat{\beta}_1(y_1 * x_{1, 1} + y_2 * x_{2, 1} + ... + y_n * x_{n, 1}) + ... + \hat{\beta}_n(y_1 * x_{1, n} + y_2 * x_{2, n} + ... + y_n * x_{n, n}))$

Suppose we take the derivative with respect to $\hat{\beta}_0$. Then we have $-(y_1 + y_2 + ... y_n)$ and every other term is a constant so $0$ in the derivative. Similarly, when we take the derivative with respect to $\hat{\beta}_1$ we get $-(y_1 * x_{1, 1} + y_2 * x_{2, 1} + ... + y_n * x_{n, 1})$ and repeating the process with respect to $\hat{\beta}_n$ gives $-(y_1 * x_{1, n} + y_2 * x_{2, n} + ... + y_n * x_{n, n})$. As can be seen below, the derivative is equal to $-X^T\vec{y}$

$-X^T\vec{y} = -(\begin{bmatrix} 1 & 1 & ... & 1 \\ x_{1, 1} & x_{2, 1} & ... & x_{n, 1} \\ ... & ... & ... & ... \\ x_{1, n} & x_{2, n} & ... & x_{n, n} \end{bmatrix} \begin{bmatrix} y_1 \\ y_2 \\ ... \\ y_n \end{bmatrix})$

Similarly $- \vec{\hat{\beta}}^T X^T \vec{y} = -(\begin{bmatrix} \hat{\beta}_0 \\ \hat{\beta}_1 \\ ... \\ \hat{\beta}_n \end{bmatrix}^T \begin{bmatrix} 1 & 1 & ... & 1 \\ x_{1, 1} & x_{2, 1} & ... & x_{n, 1} \\ ... & ... & ... & ... \\ x_{1, n} & x_{2, n} & ... & x_{n, n} \end{bmatrix}\begin{bmatrix} y_1 \\ y_2 \\ ... \\ y_n \end{bmatrix})$ and this provides the exact same sum, which can be observed by first computing $X^T \vec{y}$ and then multiplying $\vec{\hat{\beta}}^T$ to the result, which is valid as matrix vector multiplication is associative ($\vec{x}^T(A\vec{x}) = (\vec{x}^TA)\vec{x}$). Thus the derivative here is also $-X^T\vec{y}$. Our derivative

$\frac{\partial}{\partial \vec{\hat{\beta}}} (\vec{y}^T\vec{y} - \vec{y}^T X\vec{\hat{\beta}} - \vec{\hat{\beta}}^T X^T \vec{y} + \vec{\hat{\beta}}^T X^TX\vec{\hat{\beta}}) = -2X^T\vec{y} + 2X^TX\vec{\hat{\beta}}$. Now to get the estimated coefficients that minimize the $SSE$ we, as stated earlier, set this result equal to $0$ and solve

$-2X^T\vec{y} + 2X^TX\vec{\hat{\beta}} = 0, 2X^TX\vec{\hat{\beta}} = 2X^T\vec{y}, X^TX\vec{\hat{\beta}} = X^T\vec{y}, \vec{\hat{\beta}} = (X^TX)^{-1}X^T\vec{y}$

As expected this results in a $n + 1$ x $1$ vector which it should as every element is our least squares coefficient estimate for $n + 1$ coefficients. The following is a very simple example replicated in R studio to show that this is how it works.

Suppose we have the following data... $\vec{y} = \begin{bmatrix} 1 \\ 3 \\ 6 \end{bmatrix}$ and $X = \begin{bmatrix} 1 & 2 \\ 1 & 3 \\ 1 & 5 \end{bmatrix}$ meaning we have one predictor variable ($x_1$) and the ordered pairs $(2, 1), (3, 3), (5, 6)$. Our least squares expression is given by $(1 - (\hat{\beta_0} + \hat{\beta_1} * 2))^2 + (3 - (\hat{\beta_0} + \hat{\beta_1} * 3))^2 + (6 - (\hat{\beta_0} + \hat{\beta_1} * 5))^2$ which is equal to the polynomial $3x^{2} + 20xy -20x + 38y^{2} -82y + 46$ in the cartesian plane and is plotted below

enter image description here

Clearly this is a convex curve and has an absolute, global minimum around (-2, 2) in the $x, y$ plane where $x$ corresponds to $\hat{\beta_0}$ and $y$ to $\hat{\beta_1}$. In other words, negative intercept, positive slope. Let's see whether this is correct.

$\vec{\hat{\beta}} = (X^TX)^{-1}X^T\vec{y} = (\begin{bmatrix} 1 & 2 \\ 1 & 3 \\ 1 & 5 \end{bmatrix}^T \begin{bmatrix} 1 & 2 \\ 1 & 3 \\ 1 & 5 \end{bmatrix})^{-1}\begin{bmatrix} 1 & 2 \\ 1 & 3 \\ 1 & 5 \end{bmatrix}^T\begin{bmatrix} 1 \\ 3 \\ 6 \end{bmatrix} = \begin{bmatrix} 3 & 10 \\ 10 & 38 \end{bmatrix}^{-1}\begin{bmatrix} 1 & 2 \\ 1 & 3 \\ 1 & 5 \end{bmatrix}^T\begin{bmatrix} 1 \\ 3 \\ 6 \end{bmatrix} = \begin{bmatrix} \frac{19}{7} & \frac{-5}{7} \\ \frac{-5}{7} & \frac{3}{14} \end{bmatrix}\begin{bmatrix} 1 & 2 \\ 1 & 3 \\ 1 & 5 \end{bmatrix}^T\begin{bmatrix} 1 \\ 3 \\ 6 \end{bmatrix} = \begin{bmatrix} \frac{9}{7} & \frac{4}{7} & \frac{-6}{7} \\ \frac{-2}{7} & \frac{-1}{14} & \frac{5}{14} \end{bmatrix}\begin{bmatrix} 1 \\ 3 \\ 6 \end{bmatrix} = \begin{bmatrix} \frac{-15}{7} \\ \frac{23}{14} \end{bmatrix} = \begin{bmatrix} -2.14286 \\ 1.64286 \end{bmatrix} = \begin{bmatrix} \hat{\beta_0} \\ \hat{\beta_1} \end{bmatrix}$

this is pretty close to what we thought it would be when plotting the $SSE$ curve. R studio should confirm that this result is true

enter image description here

and we can see that it does. And that is how coefficients are estimated using least squares in linear regression.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.