What effect would it have on a smoothing spline to use the third (or fourth) derivative for the penalty term? Specifically, what would be the effect on the RSS if the tuning parameter were to be varied from 0 to infinity? $$ RSS=∑(y_i−f(x_i))^2 + λ∫((f(t)′′)^2dt $$
$\begingroup$
$\endgroup$
2
-
$\begingroup$ Welcome to the site, @gopal-m. I took the liberty of editing your post to use the $\LaTeX$ markdown our site affords. Please ensure it still says what you want it to. $\endgroup$– gung - Reinstate MonicaFeb 23, 2016 at 2:51
-
$\begingroup$ You'll get a higher order of spline ... $\endgroup$– Glen_bFeb 23, 2016 at 20:59
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
Let's say that we penalize the 4th derivative of $f$. This means that if $f$ is of the form $f(x) = ax^3 + bx^2 + cx + d$ then we'll have a penalty of 0. When we penalize the 2nd derivative then as $\lambda \rightarrow \infty$ we are left with the linear model that minimizes the RSS. Now as $\lambda \rightarrow \infty$ we can fit up to a cubic regression with no penalty. Certainly a cubic regression will do better than a linear model with respect to in-sample RSS so that's what we'll get. But the whole point of splines is that we avoid all of the issues that come with fitting a global polynomial (like crazy behavior near the edges of our space). So it seems to me that penalizing a higher order derivative would hamstring splines by making them at least as flexible as a polynomial regression, and by forcing a global polynomial upon us.
As $\lambda \rightarrow 0$ I don't think it matters what we penalize since there won't be a penalty either way. There is only a difference for large $\lambda$.
-
$\begingroup$ what is the basis for the assertion that $f$ will then necessarily be a 3rd degree polynomial? How are other functions ruled out? $\endgroup$– JimFeb 25, 2018 at 14:36
-
$\begingroup$ @Jim by analogy with the 2nd derivative penalty case, I'm assuming we choose our model by minimizing the objective function for $f \in C^4$ (where $C^k$ is the space of functions with $k$ continuous derivatives) so $f^{(4)}$ is continuous, which is key. As $\lambda \to \infty$ we'll want $f^{(4)} = 0$ almost everywhere so its integral is $0$, but since $f^{(4)}$ is also continuous then we have $f^{(4)} = 0$ everywhere. This means that its antiderivative $f^{(3)}$ is constant everywhere, and so forth, so $f$ is a 3rd degree polynomial in $x$ $\endgroup$– jldFeb 26, 2018 at 3:58
-
$\begingroup$ Sure, I follow you. But isn't $f$ supposed to be a piecewise 3rd degree polynomial in the smoothing spline context? (Your argument stands regardless.) $\endgroup$– JimMar 3, 2018 at 16:24
-
1$\begingroup$ @Jim i'm working on an overhaul of this answer that hopefully will thoroughly address your questions; I'll let you know when i post it. But basically here i'm just talking about the flexibility of the totally unpenalized model. In the typical setting with penalizing $f''$ we can fit a linear model with no penalty but the RSS is bad enough that it's worth adding some curvature. In this question we're penalizing $f^{(4)}$ so now we can fit a global cubic penalty-free and any spline-ness would be on top of that. $\endgroup$– jldMar 7, 2018 at 5:39
-
$\begingroup$ That sounds great, looking forward to it. Splines have my interest. $\endgroup$– JimMar 7, 2018 at 7:40