5
$\begingroup$

The title says it all. For more info: I have a dependent and independent variable $y$ and $X$. I want to fit a square spline on the data given a single knot $k$. I can do that by fitting 2 separate square functions in the areas $X\le k$ and $X>k$ with the constraint that they have to have a common point at $X=k$ by fitting (OLS) the following regression equation:

$$ y_t = \beta_0 + \beta_1 X_t + \beta_2 X_t^2 + \beta_3 (X_t-k)_+ + \beta_4(X_t-k)_+^2 + \varepsilon_t $$

where $(X-k)_+ = max(X-k, 0)$ is a truncated function.

However, the resulting fit is discontinuous at $k$, e.g: (here $k=2.5$) discont spline

To make the fit continuous at $k$ one just needs to drop the lower-order terms of the truncated function, i.e. this is the same as above without $\beta_3 (X_t-k)_+ $ (see also this video): $$ y_t = \beta_0 + \beta_1 X_t + \beta_2 X_t^2 + \beta_4(X_t-k)_+^2 + \varepsilon_t $$

Now I get a continuous function (in the first derivative) at $X=k$: cont spline

The question is: Why?

If I take the derivative of the functions before and after $k$ I get: $$f'(X)=\beta_1 + 2\beta_2 X_t + 2\beta_4 (X_t>k) $$

  • before $k$: $f'(k)=\beta_1 + 2\beta_2 k + 0 $
  • after $k$ (meaning something like $\lim_{X\to k, X > k}{(f'(X))}$: $f'(k)=\beta_1 + 2\beta_2 k + 2\beta_4k $, since $ (X_t-k)_+$ is only active if $X_t>k$.

So the two are different, one includes $2\beta_4k$ the other one doesn't. How are they the same?

$\endgroup$
8
  • 3
    $\begingroup$ Differentiation is irrelevant (and invalid, anyway). All functions $x\to (x-k)_+$ are continuous and any linear combination of continuous functions is continuous. You seem to confuse continuity with differentiability, but they are different concepts. (Differentiability does imply continuity, however.) $\endgroup$
    – whuber
    Jan 9 at 15:56
  • 1
    $\begingroup$ @whuber Interesting, would you mind expanding? A degree-d spline should have continuous derivatives up to order d-1, so differentiation seems relevant. I'm curious about why differentiation is invalid. $\endgroup$ Jan 9 at 16:10
  • 1
    $\begingroup$ @Marjolein Differentiation is not a general way to test continuity. This question incorrectly characterizes a continuous but not everywhere differentiable curve (the blue line in the image) as "discontinuous." Comparable statements about higher order differentiability can be (easily) developed through iterative integration, which is one way to construct higher-order splines. $\endgroup$
    – whuber
    Jan 9 at 17:55
  • 2
    $\begingroup$ The term $(X_t-k)_+$ is the only one that is not differentiable. All other terms have a continuous derivative. Your “$f’(k)$ after $k$” is incorrect: the derivative of $(X_t - k)_+^2$ at $k$ is zero. $\endgroup$
    – WimC
    Jan 10 at 6:11
  • $\begingroup$ @Wim On the contrary, because $(X_t-k)_+$ is not differentiable at $k,$ it doesn't have a second derivative there, either. $\endgroup$
    – whuber
    Jan 10 at 15:51

3 Answers 3

8
$\begingroup$

The prediction formula is:

$$ E(y) = \beta_0 + \beta_1 X + \beta_2 X^2 + \beta_4(X-k)_+^2 $$

Writing out the function and its derivative, both for $X < k$ and $X \geq k$, we can check whether the function and its derivative are continuous at $X = k$ (i.e., return the same value at $x = K$):

Before $k$

If $X \leq k$, the prediction function is given by:

$$ E(y|X < k) = \beta_0 + \beta_1 X + \beta_2 X^2 $$

The first derivative w.r.t. $X$ is:

$$ \beta_1 + 2 \beta_2 X $$

At $X = k$, this first derivative equals:

$$ \beta_1 + 2 \beta_2 k $$

After $k$

If $X \geq k$, the function is:

$$ E(y|X \geq k) = \beta_0 + \beta_1 X + \beta_2 X^2 + \beta_4(X-k)^2 $$

The first derivative w.r.t. $X$ is:

$$ \beta_1 + 2 \beta_2 X + 2\beta_4(X-k) $$

At $X = k$, this first derivative equals:

$$ \beta_1 + 2 \beta_2 k $$

Thus, the first derivative is continuous at $X=k$, because both first derivatives are identical at $X = k$.

$\endgroup$
3
  • 2
    $\begingroup$ Minor suggestion: differentiate $\beta_{4}(X - k)^{2}$ directly to get $2\beta_{4}(X - k)$. $\endgroup$ Jan 10 at 7:13
  • $\begingroup$ Thanks! Indeed it's "just" a differentiation error: $(\beta_4(X-k)_+^2)' = 2\beta_4X - 2\beta_4k$. I forgot the last term. For $X=k$ this is $0$, not $2\beta_4k$. :') $\endgroup$
    – PaulG
    Jan 10 at 9:20
  • $\begingroup$ @FrankHarrell Thanks, updated! $\endgroup$ Jan 10 at 16:24
6
$\begingroup$

Overall continuity of the regression spline comes from the fact that it is a polynomial of constituent parts that are all continuous functions. To demonstrate this we need to show that the shifted ramp term $(x-k)_+$ is continuous. To facilitate this analysis, for any $k \in \mathbb{R}$ we will use the shifted ramp function $f_k: \mathbb{R} \rightarrow \mathbb{R}$ defined by:

$$f_k(x) \equiv (x-k)_+ = \begin{cases} 0 & & \text{for } x < k \\[6pt] x-k & & \text{for } x \geqslant k \\[6pt] \end{cases}$$

For any values $a,x \in \mathbb{R}$ we obtain:

$$f_k(x) - f_k(a) = (x-k)_+ - (a-k)_+ = \begin{cases} 0 & & \text{for } x < k \text{ and } a < k \\[6pt] x-k & & \text{for } x \geqslant k \text{ and } a < k \\[6pt] k-a & & \text{for } x < k \text{ and } a \geqslant k \\[6pt] x-a & & \text{for } x \geqslant k \text{ and } a \geqslant k \\[6pt] \end{cases}$$

which implies that:$^\dagger$

$$\min(0,x-a) \leqslant f_k(x) - f_k(a) \leqslant \max(0,x-a).$$

We therefore have:

$$0 \leqslant |f_k(x) - f_k(a)| \leqslant |x-a|.$$

Consequently, for any value $\varepsilon > 0$ we can take $\delta(\varepsilon) = \varepsilon$ and we have:

$$0 \leqslant |f_k(x)-f_k(a)| \leqslant \varepsilon \quad \quad \quad \text{for all } 0 \leqslant |x-a| \leqslant \delta(\varepsilon).$$

This establishes that $\lim_{x \rightarrow a} f_k(x) = f_k(a)$ which means that $f_k$ is a continuous function. The remainder of what you want to demonstrate follows from the fact that polynomials of continuous functions are also continuous.


$^\dagger$ To obtain these bounds, just maximise/minimise $f_k(x) - f_k(a)$ with respect to $k$ for each case.

$\endgroup$
5
$\begingroup$

If I take the derivative of the functions before and after $k$ I get: $$f'(X)=\beta_1 + 2\beta_2 X_t + 2\beta_4 (X_t>k) $$

  • before $k$: $f'(k)=\beta_1 + 2\beta_2 k + 0 $
  • after $k$ (meaning something like $\lim_{X\to k, X > k}{(f'(X))}$: $f'(k)=\beta_1 + 2\beta_2 k + 2\beta_4k $, since $ (X_t-k)_+$ is only active if $X_t>k$.

What you did by taking the derivatives is not very clear. Especially the meaning of $(X_t>k)$ is not very clear.

I get something different instead

$$f'(X)=\beta_1 + 2\beta_2 X_t + 2\beta_4 (X_t-k)_+$$

  • before $k$: $f'(k)=\beta_1 + 2\beta_2 X_t + 0 $

  • after $k$: $f'(k)=\beta_1 + 2\beta_2 X_t + 2\beta_4(X_t-k) $

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.