Spline basis function notation to include constraint for continuity at the knots

Question

I know splines use basis functions to approximate a function locally, so that the function in one region is approximated with a weighted sum of these basis functions. Suppose the input is one-dimensional and suppose we use $h$ to indicate the basis functions and $\beta$ for the coefficients in one region we have:

$$y(x) = \sum_{i=0}^N h_i(X) \beta_i$$

For example, assuming a first order polynomial we can use a constant basis function $h_0(X) = 1$ and a first order function $h_1(X) = X$. The sum becomes:

$$y(x) = \sum_{i=0}^1 h_i(X) \beta_i =\beta_0 h_0(X)+\beta_1 h_1(X) = \beta_0+\beta_1 X $$

In the region $j$ we can approximate the function as:

$$f_j(x) = \beta_{j,0} + \beta_{j,1} X$$

and in the region $j+1$:

$$f_{j+1}(x) = \beta_{j+1,0} + \beta_{j+1,1} X$$

To force continuity at the knot (or boundary) point $\xi$:

$$\beta_{j,0} + \beta_{j,1} \xi = \beta_{j+1,0} + \beta_{j+1,1} \xi$$

In the Elements of Statistical Learning book it is said:

A more direct way to proceed in this case is to use a base that incorporates the constraints $h_2(X)=(X-\xi)_+$ where $t_+$ denotes the positive part.

I don't understand this notation and how it fits into the previous one with the sum: what does the sum becomes? Is it something like that?

$$f_j(x) = \beta_{j,0} h_0(X)+ \beta_{j,1} h_1(X)+ \beta_{j,2} h_2(X)=\\ =\beta_{j,0} + \beta_{j,1} X+ \beta_{j,2} (X-\xi)^+ = \\= \beta_{j,0} + \beta_{j,1} X+ \beta_{j,2} X -\beta_{j,2} \xi^+ $$

and in the region $j+1$:

$$f_{j+1}(x) = \beta_{j+1,0} h_0(X)+ \beta_{j+1,1} h_1(X)+ \beta_{j+1,2} h_2(X)=\\ =\beta_{j+1,0} + \beta_{j+1,1} X+ \beta_{j+1,2} (X-\xi)^+ = \\= \beta_{j+1,0} + \beta_{j+1,1} X+ \beta_{j+1,2} X -\beta_{j+1,2} \xi^+ $$

How does this expression embed the constraint?

Answer 1

Here's how I like to think of bases using functions like $h(x) = (x - \xi)^p_+$. Usually there will be a global $p$-degree polynomial and these bases can be viewed as "correcting" it, and the knots $\xi$ are where the next correction "turns on". And because $x \approx \xi$ when a new correction starts, it begins very gently and that's why things like continuity are preserved.

Here's a gif of this process for your case.

The coefficients aren't shown, but the idea is that we start with the dotted black line as our only "on" basis function, but then when we pass the first knot, where the red basis function activates, we then add in that red line but with some weight. Since we're adding two linear functions it stays linear, and initially the red line is small so we keep continuity too, but it "bends" the current function into a different one. Then when we get to the green basis function it's the same thing, it "turns on" and again corrects the current function but keeps continuity and linearity.

This also shows why this basis is terrible numerically: as we progress, we may have small outputs but they are obtained by adding and subtracting increasingly large numbers since more and more of our basis functions are "on". That's why in practice compactly supported bases like B-splines are preferred, but they are less interpretable.

This view to me makes the contraints seem fairly apparent, and I find this much more helpful than expanding a bunch of things and looking for it there. But you are correct that in general our spline will be of the form $$ y(x) = \beta_0 + \beta_1 x + \sum_{j=2}^m \beta_j h_j(x) $$ and with this particular bases many of the $h_j$ may be zero for a particular $x$.

To show mathematically why this basis keeps continuity, we need only look at an arbitrary knot since it's linear up til then. So let $\xi$ be our knot, $g(x) = ax + b$ be the line leading into the knot, and $h(x) = (x-\xi)_+$ is the basis function that's about to activate. We're considering $$ f(x) = g(x) + \beta h(x) \\ \; \\ = \begin{cases} ax + b & x < \xi \\ (a + \beta)x + (b - \xi\beta) & x \geq \xi\end{cases}. $$ These are both linear so continuity is the only issue.

As $x$ increases to $\xi$ from below we'll have $f(x) =g(x)$ since we're just adding in $0$ from $h$. And as $x$ decreases to $\xi$ we'll be adding in smaller and smaller amounts since $|x-\xi|\to 0$, so this that the left and right hand limits agree and we have continuity.

Here's the same gif but for a cubic spline. It's not nearly as dramatic now when a new basis gets activated since their effect necessarily starts more gently since the curve is twice differentiable at the knots.