Based on 'Semiparametric Regression with R' (https://link.springer.com/chapter/10.1007%2F978-1-4939-8853-2_1), a penalized spline $$ f(x)=\beta_{0}+\beta_{1} x+\sum_{k=1}^{K} u_{k}\left(x-\kappa_{k}\right)_{+} $$ leads to the penalized least squares: $$ \text { minimize } \sum_{i=1}^{n}\left\{y_{i}-f\left(x_{i}\right)\right\}^{2} \text { subject to } \sum_{k=1}^{K} u_{k}^{2} \leq C $$ for some $C>0$. Or, with the lagrangian multiplicator: $$ \operatorname{minimize}\left[\sum_{i=1}^{n}\left\{y_{i}-f\left(x_{i}\right)\right\}^{2}+\lambda \sum_{k=1}^{K} u_{k}^{2}\right]. $$ My question is: What does $u_k$ stand for? Are these equivalent to the coefficents in regression splines? And why is $u$ squared in the minimization problem?


1 Answer 1


There are two terms in those equations,

  1. a measure of the fit of the function $f()$ to the data

    $$\sum_{i=1}^n (y_i - f(x_i))^2,$$ and

  2. a measure of complexity for the function $f()$

    $$\sum_{k=1}^K u_k(x - \kappa_k)_{+}$$

The complexity of $f()$ could be measured in a number of ways, but it's helpful if it can be written in terms of the coefficients of the function. That's what the $u_k$ are, they are the estimated value of the coefficient for the $k$th basis function in the spline basis used to represent $f()$.

I don't have access right now to the reference you mention, but from what you show, the basis functions are the $(x - \kappa_k)_{+}$, where the $\kappa_k$ are the knots of the spline and we are only taking the positive part of the function. (I think this is known as a truncated power spline basis.)

The $u_k$ are the coefficients for each basis function but they can also be thought of as the weight applied to each basis function; large weights for all basis functions implies a very wiggly function because each of the basis functions has a large "effect" in order to fit the data and that large "effect" must mean the data vary a lot locally and the spline has to adapt a lot locally to achieve good fit. Small weights indicate a simpler, less wiggly function because they imply the opposite, small local effects. Why local? As we're taking only the positive part of the function $f_k = (x - \kappa_k)_{+}$, each function only operates over part of the range of $x$, the bit to the right of the $k$th knot $\kappa_k$.

The coefficients $u_k$ are squared in the first formulation because a large negative value implies a large "effect" but which would cancel out some of the positive "effects" and thus we wouldn't be measuring the complexity of the spline if we only summed the $u_k$ – I can't quite remember now why the coefficients are squared and summed (although it likely has something to do with a Euclidean norm or measure of distance somewhere in the math that proves that constraining the squared coefficients minimises a particular measure of complexity of the function $f()$) instead of summing the absolute values $\sum_{k=1}^K |u_k| \leq C$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.