3
$\begingroup$

I am curious as to what the answer to the below question is? The question specifies a modeler has a cubic spline with knots at {10, 20, 30, 50}. They realize their model is overfitting at the ends of the distribution and wants to impose an additional constraint that the curve before the first knot and after the last knot are linear, then to calculate the DOF of the new model.

My thought process is that the answer should be 4 DOF for the new natural cubic spline model. Below is how I arrived at my answer:

The cubic spline has 8 DOF: 4+(4)(4)-(4)(3)=8 where the first 4 is for the intercept, X, X^2, X^3; then add 4 terms for earch knot; subtract out the 3 constraints at each knot to account for continuity, and the first and second derivatives to be zero.

Then to make the cubic spline with 4 knots a natural cubic spline, subtract (2)(2)=4 to get 4 DOF since the interval below the lowest knot and above the largest knot need to be made linear.

I’ve been told the correct answer is 6 DOF and I don’t understand how. I’ve been told it has something to do with interior knots vs boundary knots. Can someone shed some light as to how the answer could possibly be 6? Is my answer of 4 wrong?enter image description here

$\endgroup$
1
$\begingroup$

Let there be $K$ knots $\xi_1 < \dots < \xi_K$. I'll use $x_\min$ for the smallest observed $x$ value, and $x_\max$ is analogous.

I think the answer depends on whether or not the $\xi_j$ are assumed to be interior knots, and if we want the spline to change in $[x_\min, \xi_1]$ and $[x_\max, \xi_K]$ or not (and this also depends on whether or not $x_\min < \xi_1$ and/or $\xi_K < x_\max$).

If we only care about behavior between $\xi_1$ and $\xi_K$, then we could use the following truncated power basis: $$ h_j(x) = x^j, j=0,1,2,3 $$ and $$ h_j(x) = (x-\xi_{j-3})_+^3, \hspace{5mm} j = 4,\dots, K+3 $$ leading to $K+4$ DoF. This represents four DoF coming from the global cubic that we start with and one DoF being added for every knot we pass.

Restricting this to a natural spline means we'll constrain $\beta_2 = \beta_3 = 0$ which frees up 2 DoF, and we'll need the coefficients of $x^3$ and $x^2$ to be zero when every basis is active, so this further frees up two DoF meaning there are now $K$ DoF.

In interpolation problems and smoothing splines I think this is the right way to account for DoFs, because we have every point as a knot so there aren't separate boundary knots from $x_\min$ and $x_\max$.

But if we are always thinking of the $\xi_j$ as interior knots then we would have two more regions where the spline in changing. This is where setting $\xi_0 = x_\min$ and $\xi_{K+1} = x_\max$ makes sense and really we now have $K+2$ knots and therefore the full cubic spline has $K+6$ DoF and the natural spline has $K+2$.

In Figure 7.5 of Introduction to Statistical Learning (in the question you linked) we can tell that they are using the $K+2$ DoF version because the spline is nonlinear on all of $[x_\min, x_\max]$, rather than just on $[\xi_1, \xi_3]$.


To answer the exact question: I can't tell which formulation they want but I think $6$ is more likely correct.

For a spline that is not interpolating or smoothing, which you have here, I think having $\xi_1$ and $\xi_K$ as interior knots makes sense so the $K+2 = 6$ answer makes sense.

But in the exact wording of the question they say "the curve before the first knot and after the last knot will be linear". If they mean out of the four given knots, then that means an answer of $K=4$ is correct, but I'm guessing they worded this poorly and are thinking of $x_\min$ and $x_\max$ as the first and last knots respectively, so even though at face value this suggests an answer of $4$, with this boundary knot inclusion then again the answer is $6$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ thank you for your response! the source this question is based on is James' Intro to Statistical Learning. I've linked below another thread which is the exact question/confusion I'm having between my answer and what was deemed as the correct answer. Is there a way to know in the question asked that they want us to assume implicit bounds, making the total knots 6? stats.stackexchange.com/questions/396889/… $\endgroup$ – kres901567708 Oct 28 '19 at 16:15
  • $\begingroup$ @kres901567708 I've just completely rewritten. I think in this question they are likely tacitly assuming boundary knots that are not any of the four listed and that makes $6$ correct (I was not doing this when I went through it before; I usually use smoothing splines where that doesn't make sense so I assumed no separate boundary knots but i think that was the wrong call here), but I also think the question is not clearly worded and without knowing that they likely want boundary knots I don't think $6$ vs $4$ DoF is clear. $\endgroup$ – jld Oct 28 '19 at 18:52
  • $\begingroup$ thank you! Would you say in your opinion that I have a valid case to argue the question is unclear and 4 DOF should be an acceptable answer, or is there something in the question that gives away we should be including xmin and xmax as boundary knots? $\endgroup$ – kres901567708 Oct 29 '19 at 19:43
  • $\begingroup$ @kres901567708 I think it’s unclear but as I’ve thought more about it, with these splines (ie not smoothing or interpolating) it probably is pretty much always the case that the specified knots are interior and wouldn’t be boundary knots. Like when the $\xi$ are thought of as tuning parameters, which I think is the case with a standard regression spline, they’re often put at quantiles of the data which would make them interior $\endgroup$ – jld Oct 30 '19 at 2:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.