Show that solution to cubic smoothing spline reduces to regular least squares minimization as $\lambda$ approaches infinity I am asked to show that the solution to a smoothing splines problem of the form
$$
\text{PRSS}(f,\lambda) = \sum_{i=1}^N\left[y_i-f(x_i)\right]^2 + \lambda \int f''(t)^2 dt,
$$
with
$$
f(x) = \sum_{j=1}^{N+4} \gamma_j B_j(x),
$$
where $B_j(x)$ is a natural spline,
reduces to the regular least-squares fit as $\lambda \rightarrow \infty$.
I understand that the first term disappears in the limit and that we are left with minimizing
$$
\int f''(t)^2 dt
$$
but I am not sure how to transform this into something that resembles the least squares. Could anyone please provide me a hint as to how to get started?
 A: First, the implication
$$ \int_{0}^1 |f^{\prime \prime }(t)|^2 dt \implies f^{\prime \prime}(t) = 0$$,
is false. It's only implied that $f^{\prime \prime}(t) = 0$ almost everywhere in the interval, the (0,1) interval here. To show that as $\lambda \to \infty$ we obtain a linear fit, note that the objective function consists of two parts and If $\lambda \to \infty$ the penalty term will invariably dominate the sum of squares. Hence, it will have to be $\int |\widehat{f}^{\prime \prime }(t)|^2 dt = 0$, if $\widehat{f}$ is a solution.
But now a Taylor expansion of $\widehat{f}$ of order $1$ about 0 with integral remainder shows that
$$\widehat{f}(x) =  \alpha  + \beta x + \int_{0}^1 f^{\prime \prime}(t)(x-t)_{+} dt, $$
for some $\alpha$ and $\beta$. By the Schwarz inequality, the integral can be shown to be equal to zero, hence $\widehat{f}$ will be equal to its Taylor polynomial. 
Hope this helps.
A: When $\lambda \rightarrow \infty$, $\text{PRSS}(f,\lambda)$ is
finite if and only if
$$
\int f''(t)^2 dt = 0 \implies f''(t) = 0,
$$
which is only true if $f(t)$ is a linear function of $t$. The objective is then reduced into
$$
\sum_{i=1}^N\left[y_i-\beta_0 - \beta_1x_i\right]^2,
$$
which is the objective for classical linear least-squares regression.
