1
$\begingroup$

I was considering using natural cubic splines for my prediction problem when I had a thought: In Ridge Regression, you set out to minimize the equation; \begin{equation} F(X)=\lambda\sum_i ( b^2)+ \sum_i (b^T x_i - y_i)^2. \end{equation}

Where the first term acts as a punishment for selecting beta too large.

I was thinking, could this be applied to a natural cubic spline setting in terms of number of knots? That way, you wouldn't have to manually select the number of knots and knot locations you could just train it to find it for you.

Thoughts?

Improve this question
$\endgroup$
1

3 Answers 3

Reset to default
2
$\begingroup$

If you take a look at this wiki, you'll note that smoothing splines will do what you want. They include a penalty term for the smoothness of the function which is set through a $\lambda$ parameter just like with a ridge regression. You still need to set the parameter through cross validation or similar though, even if you don't have to choose the location of the knots.

Basically the addition of this term to the objective function is what makes the splines smooth: $\lambda\int_{n_1} ^N\hat{\mu}''(x)^2dx$

Improve this answer
$\endgroup$
4
  • $\begingroup$ Smoothing splines require special model estimation software and more computation time while producing similar estimates as regression splines. $\endgroup$
    – Frank Harrell
    Nov 27, 2014 at 12:59
  • 2
    $\begingroup$ The question wasn't about whether regression splines or regularised splines produced better estimates, the question was whether there were any available methods which used regularisation to set the number of knots in a natural cubic spline. $\endgroup$
    – analystic
    Nov 28, 2014 at 5:50
  • 1
    $\begingroup$ I'd be interested in learning more about the practical differences though, do you have any simulation studies you can recommend which examine the impact of regularising the fit? $\endgroup$
    – analystic
    Nov 28, 2014 at 5:51
  • $\begingroup$ There are many papers on the subject but I don't have the references. If using a lot of knots as with smoothing splines you definitely have to used penalized estimation to get decent mean squared error. For restricted cubic splines (natural splines), which are a type of regression spline, I choose the number of knots based on prior knowledge and based on the effective sample size available. For many problems 4 knots suffices, and you get excellent agreement with penalized smoothing splines. $\endgroup$
    – Frank Harrell
    Nov 28, 2014 at 12:21
1
$\begingroup$

As detailed in my handouts at http://biostat.mc.vanderbilt.edu/rms there are many ways to come up with an estimate of how many knots should be used without using $Y$, hence there are ways to avoid having to use penalized estimation as you are forced to do with smoothing splines. Regression splines are typically much easier to deal with. The nice thing about pre-specifying the number of knots is that you know exactly how many degrees of freedom are in every hypothesis test for the model.

I place knots where the data are dense and the relationship is not known to be linear in that region. So I put knots at quantiles of $X$ and just have to worry about how many knots to use.

Improve this answer
$\endgroup$
1
  • $\begingroup$ It sounds like the OP is actually interested in something akin to smoothing splines, or penalized regression splines. $\endgroup$
    – Andrew M
    Nov 27, 2014 at 1:29
1
$\begingroup$

I could see possibly two challenges if one penalize the knots locations.

  1. It will be a difficult optimization. For simplicity, I will illustrate my point with cubic spline, instead of natural cubic spline.
    Suppose I have data points $\{(x_1, y_1), \dots, (x_n, y_n)\}$, and I have 2 knots, $\epsilon_1, \epsilon_2$, whose location I don't know. If the number of knots is not assumed to be known, then the number of columns in coefficient matrix $X$ won't be fixed, which will add some extra-layer of difficulty. (I am not sure if it's possible to solve it.) With this setup, the objective function is \begin{align} ||Y - X\beta||^2 + \lambda (\epsilon_1^2 + \epsilon_2^2) \end{align} where \begin{align} X = \begin{bmatrix}1&x_1 &x_1^2&x_1^3&(x_1-\epsilon_1)_+^3 &(x_1 -\epsilon_2)_+^3\\...\\1&x_n &x_n^2&x_n^3&(x_n-\epsilon_1)_+^3 &(x_n -\epsilon_2)_+^3\end{bmatrix} \end{align} and $\beta$ is unknown.
    This is probably a challenging optimization problem. One possible solution is to use some method similar to RBF network. Another possible solution is alternatively optimize $\beta$ and $\epsilon$.
  2. What is the meaning of penalizing the knot locations $\epsilon$? Penalizing regression ceofficients means I don't want my coefficients to be very large, whose theoretical range is $R$. What is the meaning of penalizing the knot locations? I don't want my knots to be too large? This seems strange, since the purpose of using knots is to connect some kind of discontinuity in the data. Penalizing it basically says I don't want my discontinuity point to be very large, which may or may not make sense.

Please correct me if I am wrong.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Not clear to me what was the original question. At one point I thought the OP was asking about penalizing the NUMBER of knots, not their location. $\endgroup$
    – F. Tusell
    Jul 17, 2019 at 8:28
  • $\begingroup$ That is true. But he also mentioned "you wouldn't have to manually select the number of knots and knot locations". So I assume he wants both the number of knots and locations. I don't know how to penalize the number of knots, since if number of knots changes, then dimension of both $X$ and $\beta$ changes. $\endgroup$
    – Lei Hao
    Jul 17, 2019 at 8:37
  • $\begingroup$ @F.Tusell If you happen to know some methods that can penalize the number of knots, kindly let me know. $\endgroup$
    – Lei Hao
    Jul 17, 2019 at 8:45
  • $\begingroup$ well, if you fit for instance cubic splines with $k$ pre-specified knots, you know how many degrees of freedom you are using up, as Frank Harrell pointed in another response. Then, one could think of using something like AIC. $\endgroup$
    – F. Tusell
    Jul 17, 2019 at 8:54
  • $\begingroup$ If, however, you do not only choose the number but also the location of knots, then you are somehow doing "more" to fit the data. The only way I can think of to take that into account is to use MDL (Minimum Description Length) ideas, and count the number of bits you need to specify both the number and location of knots. You might want to see one of the books of Rissanen and also the monograph of Grünwald on this topic. $\endgroup$
    – F. Tusell
    Jul 17, 2019 at 8:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.