Simplification of expression involving least squares estimators

Question

Consider the sum of squared residuals of a linear regression given by

$$ S_i(a,b) = \sum_{i=1}^n (y_i-a-bx_i)^2$$

I have to show that the optimal values of $a$ and $b$ found using the first-order conditions do indeed minimize $S_i(a,b)$. After some calculations, we find that the Hessian matrix is

$$ H = \begin{bmatrix} 2n & 2\sum_{i=1}^n x_i \\ 2\sum_{i=1}^n x_i & 2\sum_{i=1}^nx_i^2 \end{bmatrix}$$

Now, $2n > 0$. So it remains to show

$$ 4 \cdot \left(n\sum_{i=1}^n x_i^2 - (\sum_{i=1}^n x_i)^2\right) > 0 $$

Looking online I found that

$$ \left(n\sum_{i=1}^n x_i^2 - (\sum_{i=1}^n x_i)^2\right) = n \cdot \sum_{i=1}^n (x_i-\bar{x})^2 > 0$$

where $\bar{x}$ is the mean. I can't see how to go from the expression on the left to the one on the right.

I'd appreciate it if someone could explain it to me.

Answer 1

Let's go from the right to the left.

\begin{align} n\sum_{i=1}^n (x_i - \bar{x})^2 &= n\sum_{i=1}^n (x_i^2-2\bar{x} x_i +\bar{x}^2)\\&= n\sum_{i=1}^n x_i^2-2n\bar{x} \sum_{i=1}^nx_i + n^2 \bar{x}^2 \\ &= n\sum_{i=1}^n x_i^2-2n^2\bar{x} \left( \frac{\sum_{i=1}^nx_i}{n} \right) + n^2 \bar{x}^2 \\ &=n\sum_{i=1}^n x_i^2- n^2 \bar{x}^2 \\ \end{align}

I will leave the last step for you to simplify.

By the way, the conclusion should be nonnegative rather than positive.

---

Alternatively, here is an approach in terms of matrices, we want to minimize

$$L(\|A\tilde{z}-y\|^2$$ where $A$ include a column of $1$ to handle the intercept term and $\tilde{z}$ include the intercept term as well.

Differentiate it, we obtain $$\nabla_z(L)=A^T(A\tilde{x}-y)$$

and $$\nabla_z^2(L)=A^TA \succeq 0$$