Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm working on a predictive cost model where the patient's age (an integer quantity measured in years) is one of the predictor variables. A strong nonlinear relationship between age and risk of a hospital stay is evident:

enter image description here

I'm considering a penalized regression smoothing spline for patient age. According to The Elements of Statistical Learning (Hastie et al, 2009, p.151), the optimal knot placement is one knot per unique value of member age.

Given that I'm retaining age as an integer, is the penalized smoothing spline equivalent to running a ridge regression or lasso with 101 distinct age indicator variables, one per age value found in the dataset (minus one for reference)? Over parametrization is then avoided as the coefficients on each age indicator are shrunk towards zero.

share|improve this question
    
Your proposal of age-indicators + shrinkage is essentially the same thing as a smoothing spline of order 0. –  Glen_b Apr 15 at 1:51
add comment

2 Answers 2

up vote 5 down vote accepted

Great question. I believe that the answer to the question you ask - "is the penalized smoothing spline equivalent to running a ridge regression or lasso" - is yes. There are a number of sources out there that can provide commentary & perspective. One place that you may want to start with is this PDF link. As is noted in the notes:

"Fitting a smoothing spline model amounts to performing a form of ridge regression in a basis for natural splines."

If you are looking for some general reading, you might enjoy checking out this excellent paper on Penalized Regressions: The Bridge Versus the Lasso. This might help answer the question of whether the penalized smoothing spline is exactly equivalent - though it provides more general perspective. I do find it interesting as they compared different techniques to each other, specifically a new bridge regression model with the LASSO, as well as Ridge Regression.

Another more tactical place to check might be the package notes for the smooth.spline package in R. Note that they hint at the relationship here, by observing that: "with these definitions, where the B-spline basis representation can be stated as f = X c (i.e., c is the vector of spline coefficients), the penalized log likelihood is L = (y - f)' W (y - f) + λ c' Σ c, and hence c is the solution of the (ridge regression) (X' W X + λ Σ) c = X' W y."

share|improve this answer
    
Excellent, thank you Nathaniel! –  RobertF Apr 14 at 18:42
    
No worries @RobertF. Have a great afternoon. –  Nathaniel Payne Apr 15 at 20:10
add comment

I am not sure you really want so many knots, given the plot.

It looks like you may have some small samples at particular ages; the peak at 74 and the 0 values at low and high end make little sense.

Given the authority of the source you site, perhaps you want restricted cubic splines instead, with a much smaller number of knots?

share|improve this answer
1  
Thanks Peter - yes # of obs are sparse for very young and old. Using so many knots does seem counterintuitive, I did a mental double-take when first reading in ESL that placing a knot on every observation minimizes the penalized residual sum of squares. I suppose the proof is in the pudding whether a restricted cubic spline or penalized smoothing spline works better at predicting my response variable in the test dataset. –  RobertF Apr 14 at 18:55
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.