How are the penalized splines defined here?

Question

Based on 'Semiparametric Regression with R' (https://link.springer.com/chapter/10.1007%2F978-1-4939-8853-2_1), a penalized spline $$ f(x)=\beta_{0}+\beta_{1} x+\sum_{k=1}^{K} u_{k}\left(x-\kappa_{k}\right)_{+} $$ leads to the penalized least squares: $$ \text { minimize } \sum_{i=1}^{n}\left\{y_{i}-f\left(x_{i}\right)\right\}^{2} \text { subject to } \sum_{k=1}^{K} u_{k}^{2} \leq C $$ for some $C>0$. Or, with the lagrangian multiplicator: $$ \operatorname{minimize}\left[\sum_{i=1}^{n}\left\{y_{i}-f\left(x_{i}\right)\right\}^{2}+\lambda \sum_{k=1}^{K} u_{k}^{2}\right]. $$ My question is: What does $u_k$ stand for? Are these equivalent to the coefficents in regression splines? And why is $u$ squared in the minimization problem?

Answer 1

There are two terms in those equations,

1. a measure of the fit of the function $f()$ to the data

   $$\sum_{i=1}^n (y_i - f(x_i))^2,$$ and

2. a measure of complexity for the function $f()$

   $$\sum_{k=1}^K u_k(x - \kappa_k)_{+}$$

The complexity of $f()$ could be measured in a number of ways, but it's helpful if it can be written in terms of the coefficients of the function. That's what the $u_k$ are, they are the estimated value of the coefficient for the $k$th basis function in the spline basis used to represent $f()$.

I don't have access right now to the reference you mention, but from what you show, the basis functions are the $(x - \kappa_k)_{+}$, where the $\kappa_k$ are the knots of the spline and we are only taking the positive part of the function. (I think this is known as a truncated power spline basis.)

The $u_k$ are the coefficients for each basis function but they can also be thought of as the weight applied to each basis function; large weights for all basis functions implies a very wiggly function because each of the basis functions has a large "effect" in order to fit the data and that large "effect" must mean the data vary a lot locally and the spline has to adapt a lot locally to achieve good fit. Small weights indicate a simpler, less wiggly function because they imply the opposite, small local effects. Why local? As we're taking only the positive part of the function $f_k = (x - \kappa_k)_{+}$, each function only operates over part of the range of $x$, the bit to the right of the $k$th knot $\kappa_k$.

The coefficients $u_k$ are squared in the first formulation because a large negative value implies a large "effect" but which would cancel out some of the positive "effects" and thus we wouldn't be measuring the complexity of the spline if we only summed the $u_k$ – I can't quite remember now why the coefficients are squared and summed (although it likely has something to do with a Euclidean norm or measure of distance somewhere in the math that proves that constraining the squared coefficients minimises a particular measure of complexity of the function $f()$) instead of summing the absolute values $\sum_{k=1}^K |u_k| \leq C$.