11
$\begingroup$

There are words from the The Elements of Statistical Learning on page 119:

It is not hard to show that the following basis represents a cubic spline with knots at $\xi_1$ and $\xi_2$:

  • $h_1(X)=1$
  • $h_2(X)=X$
  • $h_3(X)=X^2$
  • $h_4(X)=X^3$
  • $h_5(X)=(X-\xi_1)_+^3$
  • $h_6(X)=(X-\xi_2)_+^3$

Then my question is that, according to the numeric analysis, piecewise cubic polynominals tries to fit the function in region $[\xi_1,\xi_2]$ with polynomials at order 3, so I think the basis should be

  • $h_1(X)=f(\xi_1)$
  • $h_2(X)=(X-\xi_1)$
  • $h_3(X)=(X-\xi_1)^2$
  • $h_4(X)=(X-\xi_1)^3$.

There are at most 4 basis, so where am I wrong?

$\endgroup$
5
  • 1
    $\begingroup$ Which edition/printing do you have? It's on p143 in the latest version (2nd edition, 5th printing with corrections). There's more explanation there than you give above. $\endgroup$
    – onestop
    Feb 20, 2012 at 11:18
  • $\begingroup$ I have the first edition and content about piecewise cubic polynomial in second edition are similar with the related content in first edition. I just don't understand why the cubic basis are constructed in that way. $\endgroup$
    – jerry_sjtu
    Feb 20, 2012 at 15:06
  • 2
    $\begingroup$ At the bottom, you give a basis for all cubic polynomials. Although its elements depend on (one) of the knots, the space of functions they generate is independent of the knots. Moreover, splines have more degrees of freedom than polynomials: they are allowed to change at each knot. Depending on the type of spline, this adds at least one degree of freedom for each knot. Therefore I would expect a basis of cubic splines with two knots to have a dimension of at least 4+2=6, exactly as stated in the quotation. $\endgroup$
    – whuber
    Feb 20, 2012 at 15:13
  • $\begingroup$ I don't understand the splines have more degrees of freedom than polynomials: they are allowed to change at each knot. In my opinion, cubic splines are polynomials in each region with the three conditions at the knots. @whuber $\endgroup$
    – jerry_sjtu
    Feb 20, 2012 at 15:35
  • $\begingroup$ I recommend watching the statistical learning videos at Stanford. The "Splines" videos really helped me understand them better. The text is DENSE. $\endgroup$ Jan 27, 2019 at 22:17

1 Answer 1

17
$\begingroup$

Look at a simpler problem: construct a basis for the space of piecewise constant functions whose values are allowed to break at the knots. With two knots, that's three intervals. One basis would consist of (a) the function that equals $1$ for all arguments less than or equal to $\xi_1$ and otherwise is $0$, (b) the function equal to $1$ for all arguments from $\xi_1$ through $\xi_2$ and otherwise is $0$, and (c) the function equal to $1$ for all arguments greater than $\xi_2$ but otherwise is $0$. However, there's another way. The idea is to let the basis elements encode the jumps that occur at the knots. The first basis element therefore is a constant function, say $1$, regardless of the knots. The second basis element encodes a jump at $\xi_1$. It's convenient to take it to equal $0$ for values less than or equal to $\xi_1$ and to equal $1$ for larger values. Let's call this function $H_{\xi_1}$. The third basis element can be taken to be $H_{\xi_2}$.

For example, the piecewise constant function that jumps from $48$ to $-120$ and then to $240$ at knots $\xi_1 = 2$ and $\xi_2 = 4$ can be written as $48 -168H_2 + 360H_4$: in this form, it reveals itself explicitly as a jump of $-168$ at $\xi_1=2$ followed by a jump of $+360$ at $\xi_2=4$, after starting from a baseline value of $48$.

spline 0

Here is a piecewise constant spline with two knots. It is determined by its three levels or, equivalently, by a "baseline" level and two jumps.

It should be clear that although the space of constant functions has dimension $1$, the space of piecewise constant functions with $k\ge 0$ knots has dimension $k+1$: one for a "baseline" constant plus $k$ more dimensions, one for each possible jump.

Cubic splines are obtained by integrating piecewise constant functions three times. This introduces three constants of integration. We can absorb them into the integral of the constant function. This gives a "baseline" cubic spanned by $1$, $x$, $x^2$, and $x^3$. Modulo these constants of integration, the integral of $H_{\xi}$ is $\frac{1}{3!}(x-\xi)_+^3$: its third derivative jumps by $1$ at the value $\xi$ and otherwise is constant (equal to $0$ to the left of $\xi$ and $1$ to the right of $\xi$). The basis named in the quotation merely rescales these functions by $3!$.

Spline 3

Here is a third integral of the preceding piecewise constant function. Notice that no cubic polynomial possibly can behave this way (it cannot have two flat or nearly-flat sections). Splines are inherently more flexible than polynomials of the same degree; they span a higher-dimensional space of functions.

It should now be obvious how to extend this formulation to any number of knots and to any degree of splines. Understanding the procedure can be useful when you need non-standard splines for specific problems. For instance, I recently had to develop circular quadratic splines for a regression that involved an angular independent variable (an orientation in the plane modulo $180$ degrees).

$\endgroup$
4
  • 1
    $\begingroup$ (+1) It is interesting that the "right" intuition (like this) seems to so often be far removed from a reasonable implementation. I would hope no software actually uses the basis in the OP's question. $\endgroup$
    – cardinal
    Feb 20, 2012 at 17:39
  • 1
    $\begingroup$ That's an interesting observation. I have interpreted this question in the spirit of characterizing the entire vector space rather than selecting a suitable basis for it. I think you could start with almost any basis you like and orthonormalize it before applying it to the data. In practice, I believe certain bases are chosen for their interpretability. For instance, one might replace the (potentially problematic) collection of $\{X^k\}$ by $\{(X-\mu)^k\}$ where $\mu$ is the center of the data. $\endgroup$
    – whuber
    Feb 20, 2012 at 17:47
  • $\begingroup$ If I look at $N_{i,0} = \begin{cases} 1, t\in [t_i,t_{i+1})\\ 0, \text{else}\end{cases}$ (which is a first order B-Spline) and integrate three times, I'll get $N_{i,0} =\frac{1}{6}t^3+\frac{1}{2}p_1+p_2t+p_3$ where $p_i$ are constants of integration from each step. I do not quite understand how you'll get $\frac{1}{3!}(x-\xi)^3_+$. Additionally to get a smooth, twice differentiable cublic spline function I would need to impose conditions on the first and second derivative. Looking at statistical text-books such as Wood - GAM in R, I dont quite get how a cubic spline basis actually looks like. $\endgroup$
    – Druss2k
    Jul 15, 2014 at 11:20
  • $\begingroup$ @Druss Expand $(x-\xi)_{+}^3$ using the Binomial Theorem to expose the relationship between it and the constants of integration. Re your second point: when you integrate a piecewise constant function, the result is piecewise continuously differentiable; another integration makes it piecewise twice continuously differentiable; and the third integration makes it piecewise thrice continuously differentiable. Thus, there is no reason to impose any conditions on the derivatives when constructing the splines this way. $\endgroup$
    – whuber
    Jan 26, 2019 at 18:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.