# Show that solution to cubic smoothing spline reduces to regular least squares minimization as $\lambda$ approaches infinity

I am asked to show that the solution to a smoothing splines problem of the form $$\text{PRSS}(f,\lambda) = \sum_{i=1}^N\left[y_i-f(x_i)\right]^2 + \lambda \int f''(t)^2 dt,$$ with $$f(x) = \sum_{j=1}^{N+4} \gamma_j B_j(x),$$ where $$B_j(x)$$ is a natural spline, reduces to the regular least-squares fit as $$\lambda \rightarrow \infty$$.

I understand that the first term disappears in the limit and that we are left with minimizing $$\int f''(t)^2 dt$$ but I am not sure how to transform this into something that resembles the least squares. Could anyone please provide me a hint as to how to get started?

• Since PRSS is a differentiable function of the $\gamma_j,$ it is natural to find critical values by equating the gradient of PRSS to zero.
– whuber
Sep 19, 2018 at 17:29

First, the implication

$$\int_{0}^1 |f^{\prime \prime }(t)|^2 dt \implies f^{\prime \prime}(t) = 0$$,

is false. It's only implied that $$f^{\prime \prime}(t) = 0$$ almost everywhere in the interval, the (0,1) interval here. To show that as $$\lambda \to \infty$$ we obtain a linear fit, note that the objective function consists of two parts and If $$\lambda \to \infty$$ the penalty term will invariably dominate the sum of squares. Hence, it will have to be $$\int |\widehat{f}^{\prime \prime }(t)|^2 dt = 0$$, if $$\widehat{f}$$ is a solution.

But now a Taylor expansion of $$\widehat{f}$$ of order $$1$$ about 0 with integral remainder shows that

$$\widehat{f}(x) = \alpha + \beta x + \int_{0}^1 f^{\prime \prime}(t)(x-t)_{+} dt,$$ for some $$\alpha$$ and $$\beta$$. By the Schwarz inequality, the integral can be shown to be equal to zero, hence $$\widehat{f}$$ will be equal to its Taylor polynomial.

Hope this helps.

When $$\lambda \rightarrow \infty$$, $$\text{PRSS}(f,\lambda)$$ is finite if and only if $$\int f''(t)^2 dt = 0 \implies f''(t) = 0,$$ which is only true if $$f(t)$$ is a linear function of $$t$$. The objective is then reduced into $$\sum_{i=1}^N\left[y_i-\beta_0 - \beta_1x_i\right]^2,$$ which is the objective for classical linear least-squares regression.

• This is good intuition but it lacks rigor: it is not necessarily the case that a sequence of solutions to a sequence of minimization problems will converge to a solution of the limit of the minimization problems. The objective function you write down here does not agree with the objective you wrote in your question, either.
– whuber
Sep 25, 2018 at 17:25