What is Dimension of basis in splines

Question

From mgcv package, gam(for generalized additive model fit) function uses the parameter 'k' which is dimension of basis.

Can anyone explain to me what does dimension of basis function means in spline in layman terms?

Answer 1

The dimension of the basis is the number of basis function in the basis.

Typically, k reflects how many basis functions are created initially, but identifiability constraints may lower the number of basis functions per smooth that are actually used to fit the model.

k sets some upper limit on the number of basis functions, but typically some of the basis functions will be removed when identifiability constraint are applied. For example, the default k with the default smooth type (Thin plate regression splines) is 10 for a univariate smooth. However, as one of those basis functions is a flat function, the model become unidentifiable when there is an intercept in the model; this horizontal function is the same thing as the model intercept, so you could add any value to the coef for the intercept and subtract the same value from the coef for the horizontal basis function and get the same fit but via a different model (the coefs are different). As such, the flat/horizontal basis function is removed from the basis, resulting in 9 basis functions used to fit the model.

Answer 2

The easiest way of thinking of a basis is like the building blocks or the set of prototypes used to create something more complex. So when we are setting $k$ to a low number we indirectly force a relatively simple overall shape.

Putting the notion of thing plate regression splines aside for a moment, one of the most trivial basis we use are the polynomial basis. Assuming that our predictor $x$ spans from $[-1,1]$ if we use a polynomial basis of degree 4 we have something like the following:

x = seq(-1,1, by=0.01)
matplot(x, poly(x, degree = 4, raw = TRUE), t='l', lwd = 3, 
        lty=1, panel.first=grid()); 
legend("bottomright",col = 1:4, lwd=3, legend = paste0("Degree:", 1:4))

Which means we can allow of the behaviour of our predictor $x$ to vary according to any valid linear combination of these four basis functions. Great, right? But maybe we want to ensure we have periodic behaviour. Say, our $x$ represents the day of the year of something that clearly is periodic. The previous basis shown, the polynomial one, is not constrained to "start and finish" with the same value. Good thing is that a certain mathematical giant named Joseph Fourier made the concept of representing a periodic function (or signal) into the weighted sum of sines and cosines into a field of its own (called Fourier Analysis). The important bit here is that we can thus represent a periodic signal using a series of sines and cosines and thus be certain that any combination of them will be period.

x = seq(1,365, by=1)
plot(  fda::create.fourier.basis(range(x), nbasis = 3), lwd= 3, lty=1)
grid(ny = 12)
legend("bottomright",col = 1:4, lwd=3, legend = paste0("N-th basis:", 1:3))

And we can easily see here that all basis shown will start and finish on the same value. This brings us to $k$. While the basis shown is great in terms of periodicity, we cannot represent a signal/function that oscillates more than twice in this yearly period. The "black prototype" will allows to move the baseline of the signal up or down, the "green prototype" will allows to have a slow year-long oscillation pattern and the "red prototype" will allows to have a six-month counter-balancing oscillation pattern. But what if we have monthly periodicity? Using this proposed basis with just three basis function we cannot model it. Simple as that. If we increase $k$ we can have fast oscillations, i.e. model more frequent changes. For example $k = 7$ would allows us to use a more extensive collection of building blocks like this:

(Note that this effectively what s does when we define the basis type to be a cyclic spline s(..., bs = 'cc').)

OK, so finally back to the thin-plate smoothing spline which is what mgcv::gam uses by default. The thin-plate smoothing spline estimates a smoothing function $f$ that minimizes a penalised least squares function $g$:

$g(x,y,\lambda) = \sum^N_{i=1} (y_i - f(x_i)^2) + \lambda J(f)$

where $x$ is our explanatory variable, $y$ is our response, $\lambda$ is our penalisation parameter, $N$ is the number of available data-points and $J$ is a function that penalised how complex/wiggly the function $f$ is. Now, without going to any gory details, $k$ controls the number of building blocks for $f$ and thus is another way to make the overall fit be simpler/less complex. The aptly named 2003 paper "Thin plate regression splines" by S. Wood (the lead developer of mgcv), goes through the exact mechanics of how low-rank (i.e. not very large $k$) thin plate spline can be used as smoothers. Specifically for mgcv::gam, as Gavin mentioned (+1 if you have not already), $k$ is an upper limit and not necessary what will be used.

So to conclude, the dimension of basis function means the number of building blocks we are allowed to used. Smaller number constrain us to simpler variational patterns while higher numbers allows to account for finer details. And this is the main methodological burden that GAM estimation tries to alleviate; we want to allow for the right amount of complexity; too little (too small $k$) and we under-fit missing important information, too high (too high $k$) and we over-fit finding patterns that are not really there.