What effect would it have on a smoothing spline to use the third (or fourth) derivative for the penalty term? Specifically, what would be the effect on the RSS if the tuning parameter were to be varied from 0 to infinity? $$ RSS=∑(y_i−f(x_i))^2 + λ∫((f(t)′′)^2dt $$

  • $\begingroup$ Welcome to the site, @gopal-m. I took the liberty of editing your post to use the $\LaTeX$ markdown our site affords. Please ensure it still says what you want it to. $\endgroup$ – gung - Reinstate Monica Feb 23 '16 at 2:51
  • $\begingroup$ You'll get a higher order of spline ... $\endgroup$ – Glen_b Feb 23 '16 at 20:59

Let's say that we penalize the 4th derivative of $f$. This means that if $f$ is of the form $f(x) = ax^3 + bx^2 + cx + d$ then we'll have a penalty of 0. When we penalize the 2nd derivative then as $\lambda \rightarrow \infty$ we are left with the linear model that minimizes the RSS. Now as $\lambda \rightarrow \infty$ we can fit up to a cubic regression with no penalty. Certainly a cubic regression will do better than a linear model with respect to in-sample RSS so that's what we'll get. But the whole point of splines is that we avoid all of the issues that come with fitting a global polynomial (like crazy behavior near the edges of our space). So it seems to me that penalizing a higher order derivative would hamstring splines by making them at least as flexible as a polynomial regression, and by forcing a global polynomial upon us.

As $\lambda \rightarrow 0$ I don't think it matters what we penalize since there won't be a penalty either way. There is only a difference for large $\lambda$.

| cite | improve this answer | |
  • $\begingroup$ what is the basis for the assertion that $f$ will then necessarily be a 3rd degree polynomial? How are other functions ruled out? $\endgroup$ – Jim Feb 25 '18 at 14:36
  • $\begingroup$ @Jim by analogy with the 2nd derivative penalty case, I'm assuming we choose our model by minimizing the objective function for $f \in C^4$ (where $C^k$ is the space of functions with $k$ continuous derivatives) so $f^{(4)}$ is continuous, which is key. As $\lambda \to \infty$ we'll want $f^{(4)} = 0$ almost everywhere so its integral is $0$, but since $f^{(4)}$ is also continuous then we have $f^{(4)} = 0$ everywhere. This means that its antiderivative $f^{(3)}$ is constant everywhere, and so forth, so $f$ is a 3rd degree polynomial in $x$ $\endgroup$ – jld Feb 26 '18 at 3:58
  • $\begingroup$ Sure, I follow you. But isn't $f$ supposed to be a piecewise 3rd degree polynomial in the smoothing spline context? (Your argument stands regardless.) $\endgroup$ – Jim Mar 3 '18 at 16:24
  • 1
    $\begingroup$ @Jim i'm working on an overhaul of this answer that hopefully will thoroughly address your questions; I'll let you know when i post it. But basically here i'm just talking about the flexibility of the totally unpenalized model. In the typical setting with penalizing $f''$ we can fit a linear model with no penalty but the RSS is bad enough that it's worth adding some curvature. In this question we're penalizing $f^{(4)}$ so now we can fit a global cubic penalty-free and any spline-ness would be on top of that. $\endgroup$ – jld Mar 7 '18 at 5:39
  • $\begingroup$ That sounds great, looking forward to it. Splines have my interest. $\endgroup$ – Jim Mar 7 '18 at 7:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.