'Punishment Function' in Number of Knots in Splines? I was considering using natural cubic splines for my prediction problem when I had a thought:
In Ridge Regression, you set out to minimize the equation; 
\begin{equation}
  F(X)=\lambda\sum_i ( b^2)+ \sum_i (b^T x_i - y_i)^2.
\end{equation}
Where the first term acts as a punishment for selecting beta too large. 
I was thinking, could this be applied to a natural cubic spline setting in terms of number of knots? That way, you wouldn't have to manually select the number of knots and knot locations you could just train it to find it for you.
Thoughts? 
 A: If you take a look at this wiki, you'll note that smoothing splines will do what you want. They include a penalty term for the smoothness of the function which is set through a $\lambda$ parameter just like with a ridge regression. You still need to set the parameter through cross validation or similar though, even if you don't have to choose the location of the knots.
Basically the addition of this term to the objective function is what makes the splines smooth: $\lambda\int_{n_1} ^N\hat{\mu}''(x)^2dx$
A: As detailed in my handouts at http://biostat.mc.vanderbilt.edu/rms there are many ways to come up with an estimate of how many knots should be used without using $Y$, hence there are ways to avoid having to use penalized estimation as you are forced to do with smoothing splines.  Regression splines are typically much easier to deal with.  The nice thing about pre-specifying the number of knots is that you know exactly how many degrees of freedom are in every hypothesis test for the model.
I place knots where the data are dense and the relationship is not known to be linear in that region.  So I put knots at quantiles of $X$ and just have to worry about how many knots to use.
A: I could see possibly two challenges if one penalize the knots locations.


*

*It will be a difficult optimization. For simplicity, I will illustrate my point with cubic spline, instead of natural cubic spline.
Suppose I have data points $\{(x_1, y_1), \dots, (x_n, y_n)\}$, and I have 2 knots, $\epsilon_1, \epsilon_2$, whose location I don't know. If the number of knots is not assumed to be known, then the number of columns in coefficient matrix $X$ won't be fixed, which will add some extra-layer of difficulty. (I am not sure if it's possible to solve it.) With this setup, the objective function is 
\begin{align}
 ||Y - X\beta||^2 + \lambda (\epsilon_1^2 + \epsilon_2^2) 
\end{align} where 
\begin{align}
X = \begin{bmatrix}1&x_1 &x_1^2&x_1^3&(x_1-\epsilon_1)_+^3 &(x_1 -\epsilon_2)_+^3\\...\\1&x_n &x_n^2&x_n^3&(x_n-\epsilon_1)_+^3 &(x_n -\epsilon_2)_+^3\end{bmatrix}
\end{align}
and $\beta$ is unknown.
This is probably a challenging optimization problem. One possible solution is to use some method similar to RBF network. Another possible solution is alternatively optimize $\beta$ and $\epsilon$.  

*What is the meaning of penalizing the knot locations $\epsilon$? Penalizing regression ceofficients means I don't want my coefficients to be very large, whose theoretical range is $R$. What is the meaning of penalizing the knot locations? I don't want my knots to be too large? This seems strange, since the purpose of using knots is to connect some kind of discontinuity in the data. Penalizing it basically says I don't want my discontinuity point to be very large, which may or may not make sense. 


Please correct me if I am wrong.  
