I'm learning about splines and the equation for a spline trying to predict the true function given data points is expressed as


Where $\beta_m$ is some linear coefficient, $h_m(x)$ is a basis function and $k$ is the number of regions/areas separated by knots.

I read that the least squares criterion is used to estimate the coefficients($\beta$), which makes perfect sense for evaluation. Then it goes on to say you can solve for all $\beta_m$ using matrices.

$$H = \begin{bmatrix}h_1(x_0) & ... & h_k(x_0)\\ ... & ... & ... \\ h_1(x_n) & ... & h_k(x_n)\end{bmatrix}$$

$h_i, i\in[1,k] :$ basis functions
$x_l, l\in[0,n] :$ Independent data points being approximated

$$\vec{\beta}=(H^TH)^{-1}H^T\vec{y}$$ $\vec{y}$ : Vector of response variables to each $x_l, l\in[0,n]$ so $\vec{y}=[y_0,...,y_n]$
$\vec{\beta} = [\beta_1 , ..., \beta_k]$

How does $\vec{\beta}=(H^TH)^{-1}H^T\vec{y}$ result in minimizing least square criteria?

  • 3
    $\begingroup$ Did you mean $(H^TH)^{-1}$? That is very different from $(H^TH^{-1})$. Also, you switch between $(H^TH^{-1})H^T\vec{y}$ and $H(H^TH{-1})H^T\vec{y}$. The first one is correct, and can be derived using calculus on linear transformations. $\endgroup$ – Robby the Belgian Aug 28 '20 at 6:43
  • $\begingroup$ @RobbytheBelgian the post has been corrected. Thanks for pointing out the errors $\endgroup$ – user8714896 Aug 28 '20 at 6:43
  • 3
    $\begingroup$ If you're happy to admit that for linear regression that $\hat{\beta} = (H^T H )^{-1} H^T y$, and you can also see that splines are just a linear model (albeit, ones with funny basis functions) then hopefully you can see why the same formula minimises MSE in the spline case $\endgroup$ – jcken Aug 28 '20 at 8:06
  • $\begingroup$ @jcken I'm not familiar with solving LR with this matrix form. I'm more used to $(1/n)\sum(y-\hat{f(}x_i))^2$. I don't quite understand how the matrix equation equates to MSE. $\endgroup$ – user8714896 Aug 28 '20 at 8:51
  • $\begingroup$ @user8714896 wikipedia has a good explanation of the solution to minimising the MSE via matrices. Look under the estimation section en.m.wikipedia.org/wiki/Linear_regression $\endgroup$ – jcken Aug 28 '20 at 19:12