0
$\begingroup$

I am learning the smoothing spline method. I saw that smoothing spline is a penalty term to reduce overfitting in linear regression. Given dataset {$(x_1,y_1),(x_2,y_2)..(x_n,y_n)$}So the formular such as: $$RSS=\sum(y_i-f(x_i))^2+\lambda\int((f(t)'')^2dt$$

Assume it is linear case so $$f(x_i)=ax_i+b$$ $$f(x_i)''=0$$

Is it correct. Could you explaint help me how to find second term in RSS ($\lambda\int((f(t)'')^2dt$) in linear regression case? Or give me one example?Thank you so much

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ What's the integral of $0^2$ between any pair of finite limits? $\endgroup$
    – Glen_b
    Commented May 22, 2014 at 10:46

1 Answer 1

Reset to default
1
$\begingroup$

You really need to read a book on spline functions - Schumaker, "Spline functions basic theory" is good. "Spline models for observational data" by Grace Wahba is a classic, with a fair dose of functional analysis. Lancaster and Salkauskas' "Curve and Surface Fitting" give more nuts and bolts.

But to boil it down to essentials.

  1. You don't have to find the penalty term, although it probably emerges during the algorithm. The key is to find a twice differentiable function that minimizes RSS. It turns out that f is a piecewise cubic. This is true for both the interpolation problem and the smoothing (aka regression) problem.
  2. Once it is known that the solution is a piecewise cubic (with joins at the data points), solving the spline problem becomes a basic linear problem in linear algebra. You don't actually need to evaluate the roughness penalty.
Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank sir, Because I want to reduce overfitting problem so I want to add some penalty term in Loss function. Do you know other kind of simple penalty term or regularization term to reduce overfitting problem in linear classification that published $\endgroup$
    – John
    Commented May 22, 2014 at 2:59
  • $\begingroup$ You might want to try lasso regression. The penalty term is the absolute value $int |f(t)| \, dt$. If your regression involves a lot of potential regressor variables, this penalty term has the advantage of zeroing out the less useful ones. $\endgroup$
    – Placidia
    Commented May 22, 2014 at 11:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.