I am learning about splines from the book "The Elements of Statistical Learning Data Mining, Inference, and Prediction" by Hastie et al. I found on page 145 that Natural cubic splines are linear beyond the boundary knots. There are $K$ knots, $\xi_1, \xi_2, ... \xi_K$ in the splines and the following is given about such a spline in the book.enter image description here

Question 1: How are 4 degrees of freedom freed up? I don't get this part.

Question 2: In the definition of $d_k(X)$ when $k=K$ then $d_K(X) = \frac 0 0$. What is the author trying to do in this formula? How does this help making sure that splines are linear beyond boundary knots?


2 Answers 2

  1. Let's start by considering ordinary cubic splines. They're cubic between every pair of knots and cubic outside the boundary knots. We start with 4df for the first cubic (left of the first boundary knot), and each knot adds one new parameter (because the continuity of cubic splines and derivatives and second derivatives adds three constraints, leaving one free parameter), making a total of $K+4$ parameters for $K$ knots.

    A natural cubic spline is linear at both ends. This constrains the cubic and quadratic parts there to 0, each reducing the df by 1. That's 2 df at each of two ends of the curve, reducing $K+4$ to $K$.

    Imagine you decide you can spend some total number of degrees of freedom ($p$, say) on your non-parametric curve estimate. Since imposing a natural spline uses 4 fewer degrees of freedom than an ordinary cubic spline (for the same number of knots), with those $p$ parameters you can have 4 more knots (and so 4 more parameters) to model the curve between the boundary knots.

  2. Note that the definition for $N_{k+2}$ is for $k=1,2,...,K-2$ (since there are $K$ basis functions in all). So the last basis function in that list, $N_{K}=d_{K-2}-d_{K-1}$. So the highest $k$ needed for definitions of $d_k$ is for $k=K-1$. (That is, we don't need to try to figure out what some $d_K$ might do, since we don't use it.)

  • $\begingroup$ Could someone explain what $d_{k}$ actually is? I am very confused by this part, and how it is used to create some of the basis functions $\endgroup$
    – Sean
    Commented May 30, 2020 at 20:37
  • $\begingroup$ @Sean - You could actually represent everything in the power-series form of $b_j(x)=(x-\xi_j)_+^3$ (this is given as exercise 5.4). the $d$'s are a new representation of the basis functions, such that the 3rd degree (cubic term) will be eliminated past the last knot. The $N$'s then make sure that the 2nd degree (quadratic term) will also be eliminated past the last knot. You need this to make sure this is a Natural Cubic Spline. $\endgroup$ Commented Jun 11 at 9:46

I detail the assertion: "This frees up four degrees of freedom (two constraints each in both boundary regions)" in an example with $2$ knots $\xi_1, \xi_2$. The related intervals are $]-\infty, \xi_1[$, $]\xi_1, \xi_2[$ and $]\xi_2, +\infty[$ (so there are $|I|=3$ intervals and $|I|-1=2$ knots).

For (common) cubic splines

Without regularity constraints, we have $4|I|=12$ equations:

$$\mathbf{1}(X < \xi_1)~~;~~\mathbf{1}(X < \xi_1)X~~;~~\mathbf{1}(X < \xi_1)X^2~~;~~\mathbf{1}(X < \xi_1)X^3~~;$$ $$\mathbf{1}(\xi_1 \leq X < \xi_2)~~;~~\mathbf{1}(\xi_1 \leq X < \xi_2)X~~;~~\mathbf{1}(\xi_1 \leq X < \xi_2)X^2~~;~~\mathbf{1}(\xi_1 \leq X < \xi_2)X^3~~;$$ $$\mathbf{1}(\xi_2 \leq X)~~;~~\mathbf{1}(\xi_2 \leq X)X~~;~~\mathbf{1}(\xi_2 \leq X)X^2~~;~~\mathbf{1}(\xi_2 \leq X)X^3.$$

By adding the constraints (cubic splines assumes a $\mathcal{C}^r$ regularity with $r=2$ ), we need to add $(r+1)\times(|I|-1) = 3\times(|I|-1) = 6$ constraints on the linear coefficients.

We end up with $12-6=6$ degree of freedom.

For natural cubic splines

"A natural cubic splines adds additional constraints, namely that function is linear beyond the boundary knots."

Without regularity constraints, we have $4|I|-4=12-4$ equations (we have removed $4$ equations, $2$ each in both boundary regions because they involve quadratic and cubic polynomials):

$$\mathbf{1}(X < \xi_1)~~;~~\mathbf{1}(X < \xi_1)X~~;~~$$ $$\mathbf{1}(\xi_1 \leq X < \xi_2)~~;~~\mathbf{1}(\xi_1 \leq X < \xi_2)X~~;~~\mathbf{1}(\xi_1 \leq X < \xi_2)X^2~~;~~\mathbf{1}(\xi_1 \leq X < \xi_2)X^3~~;$$ $$\mathbf{1}(\xi_2 \leq X)~~;~~\mathbf{1}(\xi_2 \leq X)X.$$

The constraints are the same as before, so we still need to add $3\times(|I|-1) = 6$ constraints on the linear coefficients.

We end up with $8-6=2$ degree of freedom.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.