I am asked to show that the solution to a smoothing splines problem of the form $$ \text{PRSS}(f,\lambda) = \sum_{i=1}^N\left[y_i-f(x_i)\right]^2 + \lambda \int f''(t)^2 dt, $$ with $$ f(x) = \sum_{j=1}^{N+4} \gamma_j B_j(x), $$ where $B_j(x)$ is a natural spline, reduces to the regular least-squares fit as $\lambda \rightarrow \infty$.

I understand that the first term disappears in the limit and that we are left with minimizing $$ \int f''(t)^2 dt $$ but I am not sure how to transform this into something that resembles the least squares. Could anyone please provide me a hint as to how to get started?

  • $\begingroup$ Since PRSS is a differentiable function of the $\gamma_j,$ it is natural to find critical values by equating the gradient of PRSS to zero. $\endgroup$ – whuber Sep 19 '18 at 17:29

When $\lambda \rightarrow \infty$, $\text{PRSS}(f,\lambda)$ is finite if and only if $$ \int f''(t)^2 dt = 0 \implies f''(t) = 0, $$ which is only true if $f(t)$ is a linear function of $t$. The objective is then reduced into $$ \sum_{i=1}^N\left[y_i-\beta_0 - \beta_1x_i\right]^2, $$ which is the objective for classical linear least-squares regression.

  • 1
    $\begingroup$ This is good intuition but it lacks rigor: it is not necessarily the case that a sequence of solutions to a sequence of minimization problems will converge to a solution of the limit of the minimization problems. The objective function you write down here does not agree with the objective you wrote in your question, either. $\endgroup$ – whuber Sep 25 '18 at 17:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.