4
$\begingroup$

I recently came across this tutorial on General Additive Models (GAMs). Quoting the article:

The principle behind GAMs is similar to that of regression, except that instead of summing effects of individual predictors, GAMs are a sum of smooth functions. Functions allow us to model more complex patterns, and they can be averaged to obtain smoothed curves that are more generalizable.

Is a GLM with smooth predictors (i.e., independent variables) practically any different from a GLM with smooth functions? Does smoothness actually matter?

The most competitive data scientists (e.g., those on Kaggle) say that when everyone is competing with the same modeling packages, what sets the best data scientists apart from the rest is the ability to do feature engineering well. Quoting this article:

With the extensive amount of free tools and libraries available for data analysis, everybody has the possibility of trying out advanced statistical models in a competition. As a consequence of this, what gives you most “bang for the buck” is rarely the statistical method you apply, but rather the features you apply it to. By feature engineering, I mean using domain specific knowledge or automatic methods for generating, extracting, removing or altering features in the data set.

Back to the original article, the term "function" sounds vague. To someone without applied GAM experience, such as myself, "function" sounds like a synonym for a predictor that has simply gone through feature engineering or transformation. This leads me to believe that if I am already doing feature engineering (and if I'm also using GLMs) for a predictive model, then I likely won't get any additional predictive benefit using GAMs. Is that a fair assumption?

$\endgroup$
3
  • 1
    $\begingroup$ What do you mean by "smooth predictors" in your second sentence? In a sense you are doing some aspects of feature engineering with a GAM. Fitting a GAM would be similar to including $x$, $x^2$, and $x^3$, exact than instead of representing the effect of $x$ as a set of global polynomial basis functions, the GAM uses local spline basis functions of some description. I usually takle feature engineering to mean much more than this though. GAMs are only useful is the truth is smooth or a smooth function is a close approximation to the the truth. If the truth is not smooth other features are needed $\endgroup$ Sep 13, 2017 at 15:46
  • $\begingroup$ @GavinSimpson I mean smooth only in the sense that GAM functions are smooth, as mentioned in the first link. So is it fair to say that if all of my feature engineering were simply polynomial transformations (generally not the case, but putting that aside for the moment), then the GAM and GLM would probably output similar predictions? $\endgroup$
    – Ryan Zotti
    Sep 13, 2017 at 16:34
  • 4
    $\begingroup$ Yes; a GAM is just a GLM but using a spline basis expansion for one or more of the covariates. If you took the same basis expansion and used it in the GLM you'd get the same model (for fixed degree of smoothness). The main difference comes if you want to do smoothness selection (decide how wiggly any smooths should be) as the GAM then becomes a penalized GLM and the IRLS algorithm becomes a penalized IRLS algorithm. If you fix the wiggliness, then GAM == GLM. $\endgroup$ Sep 13, 2017 at 17:13

1 Answer 1

1
$\begingroup$

In terms of predictive performance, you could get the same result by engineering the predictors of a GAM model and then using a GLM. Although for a complex non-linear relationship between the target and predictor, the best way to engineer the feature for a GLM to match the performance of a GAM, would probably be to generate a GAM for that feature.

However, I do not use GAMs primarily for predictive performance. I find the utility of GAMs is to determine the non-linear relationship between your dependent variable and a predictor. This relationship, and the confidence intervals, are more often than not all I want from the model. I can then make a useful visualisation of the results that help me explain the data.

Practically the answer to your question is; if the relationship is simple then a simple transformation and GLM would probably perform just as well; if the relationship is complex then it would be hard to engineer the feature as well as GAM does; but if your primary goal is predictive performance, then just use gradient boosting or a neural network.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy