When to use non-parametric regression such as kernel, generalized additive model, spline, and polynomial?

Question

I understand that kernel regression is a form of non-linear/non-parametric regression. However, I know you can also use generalized additive models for non-linear regression, as well as polynomials and spline.

I think I understand that the shortcomings of spline include you have to define the knots and where they are, whereas polynomials you have to define the polynomial ordering and that data doesn't always fit polynomial terms. But when do you use kernel vs. generalized additive models for nonlinear regression? Under what conditions is one better or the other more well suited? When do you overfit with non-parametric terms? I imagine this is a risk with data as it can fit the data with as many bends as the data could fit.

At a cursory glance, kernel and generalized additive model seem similar in terms of the type of data it tries to represent or analyze. When do you use on or the other?

Answer 1

There are no standard criteria for taking your choice. As a general tip, you should use the simplest useful model (this sounds great and makes me feel very good about myself, but translating it into practice is a bit harder)

My recommendation would be to work with diagnose/validation procedures that help you distinguish between useful and rubbish models. You may also want to begin with some descriptive analysis to figure out the "structure" of your data. Then try a simple model like linear regression and try to understand the reasons why it's incomplete.

Finally, keep cross-validation in mind!