1
$\begingroup$

I know splines use basis functions to approximate a function locally, so that the function in one region is approximated with a weighted sum of these basis functions. Suppose the input is one-dimensional and suppose we use $h$ to indicate the basis functions and $\beta$ for the coefficients in one region we have:

$$y(x) = \sum_{i=0}^N h_i(X) \beta_i$$

For example, assuming a first order polynomial we can use a constant basis function $h_0(X) = 1$ and a first order function $h_1(X) = X$. The sum becomes:

$$y(x) = \sum_{i=0}^1 h_i(X) \beta_i =\beta_0 h_0(X)+\beta_1 h_1(X) = \beta_0+\beta_1 X $$

In the region $j$ we can approximate the function as:

$$f_j(x) = \beta_{j,0} + \beta_{j,1} X$$

and in the region $j+1$:

$$f_{j+1}(x) = \beta_{j+1,0} + \beta_{j+1,1} X$$

To force continuity at the knot (or boundary) point $\xi$:

$$\beta_{j,0} + \beta_{j,1} \xi = \beta_{j+1,0} + \beta_{j+1,1} \xi$$

In the Elements of Statistical Learning book it is said:

A more direct way to proceed in this case is to use a base that incorporates the constraints $h_2(X)=(X-\xi)_+$ where $t_+$ denotes the positive part.

I don't understand this notation and how it fits into the previous one with the sum: what does the sum becomes? Is it something like that?

$$f_j(x) = \beta_{j,0} h_0(X)+ \beta_{j,1} h_1(X)+ \beta_{j,2} h_2(X)=\\ =\beta_{j,0} + \beta_{j,1} X+ \beta_{j,2} (X-\xi)^+ = \\= \beta_{j,0} + \beta_{j,1} X+ \beta_{j,2} X -\beta_{j,2} \xi^+ $$

and in the region $j+1$:

$$f_{j+1}(x) = \beta_{j+1,0} h_0(X)+ \beta_{j+1,1} h_1(X)+ \beta_{j+1,2} h_2(X)=\\ =\beta_{j+1,0} + \beta_{j+1,1} X+ \beta_{j+1,2} (X-\xi)^+ = \\= \beta_{j+1,0} + \beta_{j+1,1} X+ \beta_{j+1,2} X -\beta_{j+1,2} \xi^+ $$

How does this expression embed the constraint?

$\endgroup$
1
$\begingroup$

Here's how I like to think of bases using functions like $h(x) = (x - \xi)^p_+$. Usually there will be a global $p$-degree polynomial and these bases can be viewed as "correcting" it, and the knots $\xi$ are where the next correction "turns on". And because $x \approx \xi$ when a new correction starts, it begins very gently and that's why things like continuity are preserved.

Here's a gif of this process for your case.

spline

The coefficients aren't shown, but the idea is that we start with the dotted black line as our only "on" basis function, but then when we pass the first knot, where the red basis function activates, we then add in that red line but with some weight. Since we're adding two linear functions it stays linear, and initially the red line is small so we keep continuity too, but it "bends" the current function into a different one. Then when we get to the green basis function it's the same thing, it "turns on" and again corrects the current function but keeps continuity and linearity.

This also shows why this basis is terrible numerically: as we progress, we may have small outputs but they are obtained by adding and subtracting increasingly large numbers since more and more of our basis functions are "on". That's why in practice compactly supported bases like B-splines are preferred, but they are less interpretable.

This view to me makes the contraints seem fairly apparent, and I find this much more helpful than expanding a bunch of things and looking for it there. But you are correct that in general our spline will be of the form $$ y(x) = \beta_0 + \beta_1 x + \sum_{j=2}^m \beta_j h_j(x) $$ and with this particular bases many of the $h_j$ may be zero for a particular $x$.

To show mathematically why this basis keeps continuity, we need only look at an arbitrary knot since it's linear up til then. So let $\xi$ be our knot, $g(x) = ax + b$ be the line leading into the knot, and $h(x) = (x-\xi)_+$ is the basis function that's about to activate. We're considering $$ f(x) = g(x) + \beta h(x) \\ \; \\ = \begin{cases} ax + b & x < \xi \\ (a + \beta)x + (b - \xi\beta) & x \geq \xi\end{cases}. $$ These are both linear so continuity is the only issue.

As $x$ increases to $\xi$ from below we'll have $f(x) =g(x)$ since we're just adding in $0$ from $h$. And as $x$ decreases to $\xi$ we'll be adding in smaller and smaller amounts since $|x-\xi|\to 0$, so this that the left and right hand limits agree and we have continuity.

Here's the same gif but for a cubic spline. It's not nearly as dramatic now when a new basis gets activated since their effect necessarily starts more gently since the curve is twice differentiable at the knots.

enter image description here

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you the gif are really good. My bad was that I was missing the meaning of + in the basis function. Anyway lots of insights in ur question :) $\endgroup$ – Francesco Boi Oct 29 '19 at 16:23
  • $\begingroup$ @FrancescoBoi great, glad this was helpful! $\endgroup$ – jld Oct 31 '19 at 15:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.