2
$\begingroup$

Could someone tell me what it is called if you linear combine a non-linear transformation such as:

$$y_i = \beta_1 f(x_{1i}) + \beta_2 f(x_{2i}) + \ldots + \beta_n f(x_{ni}),$$

where $f(\cdot)$ is a non-linear transformation relating the predictor to the response variable? To me this seems similar to a Generalized Additive Model, but instead of having the "link", the "smoothed" expressions are provided.

$\endgroup$
3
  • $\begingroup$ Are you giving the nonlinear transformations, say sine or squaring? $\endgroup$
    – Dave
    Sep 16, 2019 at 2:41
  • $\begingroup$ I call this a simple nonlinear transformation of the raw data. It is fair game, and it doesn't affect the linearity of the model. Note that any one predictor variable may be transformed nonlinearly in any number of ways (polynomial, logarithmic, sinusoidal) and the linear superpositions of these transformations remains a linear model. A generalized additive model is a bit different. The LHS is not $Y$ but $g(E(Y|f(x_1)...f(x_n)))$, which is transformation of the expectation of $Y$ given all of the $x$'s and their transformations. $\endgroup$ Sep 16, 2019 at 2:45
  • 2
    $\begingroup$ What you describe is just ordinary multiple regression with transformed x-variables. As far as I know, it doesn't have a special name. If you want to search for references to it in a database such as Google Scholar, the best keywords might be "transformation" and "x-variable" or "covariate" or "independent variable". $\endgroup$ Sep 16, 2019 at 4:20

1 Answer 1

1
$\begingroup$

If you specify the $f$ (or $f$s), this is called feature engineering, and then you are doing linear regression on nonlinear basis functions.

When you present the data to a linear regression, it does not know or care how you got the features. If they are raw measurements, they're just numbers to be thrown into the calculation. If you were thoughtful about how to use features and decided that you should be squaring a feature, those squared values are just numbers to be thrown into the calculation. Your interpretation of the regression will depend on what you did, but your $f(x_i)$ features just as easily could be defined as $z_i$ for a regression equation of:

$$\hat y_i = \hat \beta_1 z_{1i} + \hat \beta_2 z_{2i} + \ldots + \hat \beta_n z_{ni}$$

Jeffrey Miller, known as MathematicalMonk on YouTube, has a good video about this. You don't have to use the same $f$ for each $x$ variable, and you can have basis functions of multiple variables (such as interactions) in your linear combination.

$\endgroup$
3
  • $\begingroup$ This is a rather special case of feature engineering in which all the features must be transformed in exactly the same way. Sometimes that makes sense, such as when all the features have similar meanings and are measured in the same way (think constituents of a chemical mixture, for instance), but more often than not it causes confusion among statistical neophytes who wonder whether it's necessary to use the same transformation of all their variables. And that still leaves open the possible interpretation of the question in which $f$ is to be estimated from the data. $\endgroup$
    – whuber
    May 2, 2022 at 19:26
  • $\begingroup$ @whuber the "smoothed" expressions are provided tells me that the $f$ is given, not estimated from the data. $\endgroup$
    – Dave
    May 2, 2022 at 19:29
  • $\begingroup$ I agree, but the question gives contradictory hints: the comparison to a GAM strongly suggests $f$ is to be estimated. $\endgroup$
    – whuber
    May 2, 2022 at 19:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.