2
$\begingroup$

In linear regression, the independent variables have an additive effect on the response (level-level regression):

$y=\beta_0+\beta_1x+\epsilon$

In a log-level regression, the independent variables have an additive effect on the log-transformed response and a multiplicative effect on the original untransformed response:

$log(y)=\beta_0+\beta_1x+\epsilon$

Suppose that I know for each predictor if it has an additive or multiplicative effect on the response and that I need to estimate these effects through ordinary least squares. How can I specify the formula of the model so that I estimate correctly these effects?

$\endgroup$
3
  • 1
    $\begingroup$ The two models you posit are inconsistent in how they treat the random error: do you want that to be multiplicative or additive? For instance, if it's additive then your model will be of the form $$y = \exp(\alpha_0 + \alpha_1 x_1 + \cdots + \alpha_p x_p) + \beta_0 + \beta_1 z_1 + \cdots + \beta_q z_q + \varepsilon,$$ exhibiting multiplicative effects in the $x_i$ and additive effects in the $z_j.$ $\endgroup$ – whuber Sep 23 '20 at 17:01
  • $\begingroup$ @whuber would it be possible to do something like $y = exp(\alpha_0 + \alpha_1 x_1 + \cdots + \alpha_p x_p) + \beta_0 + \beta_1 x_1 + \cdots + \beta_q x_q + \epsilon$? Or, include a multiplicative error as well as kind of a mixed model? $\endgroup$ – abalter Sep 23 '20 at 17:45
  • 2
    $\begingroup$ @Abalter With some care, yes -- although it might be difficult to tease out the separate error terms. I think some extremely specific (and unusual) assumptions about their joint distributions would be required. $\endgroup$ – whuber Sep 23 '20 at 18:15
0
$\begingroup$

You can use Linear Regression to model any linear/non-linear relationship using basis expansion (slides from Elements of Statistical Learning). In your case you could probably exponentiate some of the variables, but it might be preferable to use an automatic method, such as Multivariate Adaptive Regression Splines, that still provides interpretable results.

$\endgroup$
6
  • $\begingroup$ in the top answer to the following unrelated question, a linear regression plot (second graph) shows a non-linear regression line shown in red. how can a linear regression model produce a non-linear curve for a regression line when we know that regression lines from a linear regression model can only be straight? stats.stackexchange.com/questions/487005/… $\endgroup$ – develarist Sep 23 '20 at 16:57
  • $\begingroup$ What basis expansion of the predictors (not the outcome) would allow you to model a multiplicative effect? $\endgroup$ – Eoin Sep 23 '20 at 17:00
  • 1
    $\begingroup$ @develarist you can perform linear regression against a nonlinear function of the explanatory variables. If the real, but unknown natural relationship is something like $y = a + b \cdot sin(x) + c \cdot cos(x^2)$, then you can perform linear regression on a formula such as that. $\endgroup$ – abalter Sep 23 '20 at 17:42
  • 2
    $\begingroup$ @Devel I explain this phenomenon, with illustrations, at stats.stackexchange.com/a/354256/919. The short answer is, when the regressor variables are non-linearly related, then a linear function of those variables typically looks like a nonlinear function of any proper subset of those variables. $\endgroup$ – whuber Sep 23 '20 at 18:13
  • 1
    $\begingroup$ The ordinary least squares model doesn't care, in any way, whether the second regressor variable $x^2$ is a function of $x.$ Indeed, in old software one often had to process the data to create a separate column to hold the values of $x^2$ and then perform a multiple regression of the $y$ values against the $x$ values and the values in the second column. This little anecdote might help clarify the boundary between conducting a statistical procedure (such as OLS regression) and interpreting its output: only at the latter stage will you account for the relation between the regressors. $\endgroup$ – whuber Sep 23 '20 at 19:38
0
$\begingroup$

I don't believe it's possible in general to do this with ordinary least squares, since OLS is at heart a trick to calculate $\hat \beta$ in $E[y] = \bf{X}\hat\beta$ using matrix division.

It can be done more generally, though.

I think the tricky bit is figuring out exactly what you mean by each predictor having an additive or multiplicative effect on the response. For example, with two predictors, do you mean:

$$ y = (\beta_0 \times \beta_2 x_2) + \beta_1 x_1 ?\\ y = (\beta_0 + \beta_1 x_1) \times \beta_2 x_2 ?\\ y = \beta_0 + (\beta_1 x_1 \times \beta_2 x_2) ? \\ $$

...and there's probably others as well. Of these, the first (multiplication before applying the additive effects) is the simplest to estimate, as it has fewer high-order multiplicative terms, and is more likely to correspond to the model you intended.

Unfortunately, even this isn't simple to estimate, since the predictions with $\beta_0 = 2, \beta_2 = 2$$y = (2 \times 2 \times x_2) + \beta_1 x_1$ are the same as those when
$\beta_0 = 1, \beta_2 = 4$$y = (1 \times 4 \times x_2) + \beta_1 x_1$.

The best way around this is to use a Bayesian estimation tool like Stan to set reasonable priors on your model parameters (for example that the multiplicative effect, $\beta_2$, should be close to 1), and find the best parameter estimates that are consistent with those priors.

$\endgroup$
4
  • 1
    $\begingroup$ You are considering additive models with interactions. The usual meaning of a "multiplicative effect" is that the model satisfies a partial differential equation $\partial y/\partial x_i = \alpha_i y.$ $\endgroup$ – whuber Sep 23 '20 at 17:02
  • $\begingroup$ I'm not familiar with this framing, thanks. Does the first model not satisfy this, since $\partial y/\partial x_2 = \beta_2 y$? $\endgroup$ – Eoin Sep 23 '20 at 17:11
  • 1
    $\begingroup$ In your first model $\partial y/\partial x_2 = \beta_0\beta_2$ is not a constant multiple of $y.$ In fact, all three of your models are standard regression models but have been overparameterized; none of them is a multiplicative effects model. $\endgroup$ – whuber Sep 23 '20 at 17:17
  • 1
    $\begingroup$ Fair enough. I'll come back to this (and amend or remove my answer) tomorrow. $\endgroup$ – Eoin Sep 23 '20 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.