9
$\begingroup$

I'm working on a predictive cost model where the patient's age (an integer quantity measured in years) is one of the predictor variables. A strong nonlinear relationship between age and risk of a hospital stay is evident:

enter image description here

I'm considering a penalized regression smoothing spline for patient age. According to The Elements of Statistical Learning (Hastie et al, 2009, p.151), the optimal knot placement is one knot per unique value of member age.

Given that I'm retaining age as an integer, is the penalized smoothing spline equivalent to running a ridge regression or lasso with 101 distinct age indicator variables, one per age value found in the dataset (minus one for reference)? Over parametrization is then avoided as the coefficients on each age indicator are shrunk towards zero.

Improve this question
$\endgroup$
2
  • $\begingroup$ Your proposal of age-indicators + shrinkage is essentially the same thing as a smoothing spline of order 0. $\endgroup$
    – Glen_b
    Apr 15, 2014 at 1:51
  • $\begingroup$ It would be helpful if you specified what the other predictive variables are, as one of the answers suggested, if you control for the admission reason you might have a very different graph. $\endgroup$
    – seanv507
    Oct 27, 2016 at 15:44

3 Answers 3

Reset to default
11
$\begingroup$

Great question. I believe that the answer to the question you ask - "is the penalized smoothing spline equivalent to running a ridge regression or lasso" - is yes. There are a number of sources out there that can provide commentary & perspective. One place that you may want to start with is this PDF link. As is noted in the notes:

"Fitting a smoothing spline model amounts to performing a form of ridge regression in a basis for natural splines."

If you are looking for some general reading, you might enjoy checking out this excellent paper on Penalized Regressions: The Bridge Versus the Lasso. This might help answer the question of whether the penalized smoothing spline is exactly equivalent - though it provides more general perspective. I do find it interesting as they compared different techniques to each other, specifically a new bridge regression model with the LASSO, as well as Ridge Regression.

Another more tactical place to check might be the package notes for the smooth.spline package in R. Note that they hint at the relationship here, by observing that: "with these definitions, where the B-spline basis representation can be stated as f = X c (i.e., c is the vector of spline coefficients), the penalized log likelihood is $L = (y - f)^T W (y - f) + \lambda c^T \Sigma c$, and hence $c$ is the solution of the (ridge regression) $(X^T W X + \lambda \Sigma) c = X^T W y$."

Improve this answer
$\endgroup$
2
  • $\begingroup$ No worries @RobertF. Have a great afternoon. $\endgroup$
    – Nathaniel Payne
    Apr 15, 2014 at 20:10
  • 1
    $\begingroup$ The link to the link PDF in paragraph 1 is broken. $\endgroup$
    – Jthorpe
    Dec 4, 2017 at 20:58
3
$\begingroup$

I am not sure you really want so many knots, given the plot.

It looks like you may have some small samples at particular ages; the peak at 74 and the 0 values at low and high end make little sense.

Given the authority of the source you site, perhaps you want restricted cubic splines instead, with a much smaller number of knots?

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thanks Peter - yes # of obs are sparse for very young and old. Using so many knots does seem counterintuitive, I did a mental double-take when first reading in ESL that placing a knot on every observation minimizes the penalized residual sum of squares. I suppose the proof is in the pudding whether a restricted cubic spline or penalized smoothing spline works better at predicting my response variable in the test dataset. $\endgroup$
    – RobertF
    Apr 14, 2014 at 18:55
0
$\begingroup$

I'm late to this discussion, but look at the chart of the data ... that apparent spikeyness in the data over age 70 isn't a true reflection of age-related risk, it's a symptom of sparse data and some randomness.

You would not want to model that using one-knot-per-year, that would certainly lead to overfitting the noise.

Also, you're going to find a very different pattern if you look at female vs male. Most of the peak in the age 15-30 range is going to be Obstetrics.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi Doug - Right, there are definitely fewer observations over age 70. A penalized one year per knot spline model would likely drop the 70+ coefficients to zero. The goal here would be to replace the manual selection of knot placement with an automated process that best fits the nonlinear relationship between age and IP admits, especially useful in a predictive model. $\endgroup$
    – RobertF
    Oct 27, 2016 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.