5
$\begingroup$

In linear regression, the independent variables have an additive effect on the response (level-level regression):

$y=\beta_0+\beta_1x+\epsilon$

In a log-level regression, the independent variables have an additive effect on the log-transformed response and a multiplicative effect on the original untransformed response:

$log(y)=\beta_0+\beta_1x+\epsilon$

Suppose that I know for each predictor if it has an additive or multiplicative effect on the response and that I need to estimate these effects through ordinary least squares. How can I specify the formula of the model so that I estimate correctly these effects?

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ The two models you posit are inconsistent in how they treat the random error: do you want that to be multiplicative or additive? For instance, if it's additive then your model will be of the form $$y = \exp(\alpha_0 + \alpha_1 x_1 + \cdots + \alpha_p x_p) + \beta_0 + \beta_1 z_1 + \cdots + \beta_q z_q + \varepsilon,$$ exhibiting multiplicative effects in the $x_i$ and additive effects in the $z_j.$ $\endgroup$
    – whuber ♦
    Sep 23, 2020 at 17:01
  • $\begingroup$ @whuber would it be possible to do something like $y = exp(\alpha_0 + \alpha_1 x_1 + \cdots + \alpha_p x_p) + \beta_0 + \beta_1 x_1 + \cdots + \beta_q x_q + \epsilon$? Or, include a multiplicative error as well as kind of a mixed model? $\endgroup$
    – abalter
    Sep 23, 2020 at 17:45
  • 2
    $\begingroup$ @Abalter With some care, yes -- although it might be difficult to tease out the separate error terms. I think some extremely specific (and unusual) assumptions about their joint distributions would be required. $\endgroup$
    – whuber ♦
    Sep 23, 2020 at 18:15

2 Answers 2

Reset to default
0
$\begingroup$

You can use Linear Regression to model any linear/non-linear relationship using basis expansion (slides from Elements of Statistical Learning). In your case you could probably exponentiate some of the variables, but it might be preferable to use an automatic method, such as Multivariate Adaptive Regression Splines, that still provides interpretable results.

Improve this answer
$\endgroup$
6
  • $\begingroup$ in the top answer to the following unrelated question, a linear regression plot (second graph) shows a non-linear regression line shown in red. how can a linear regression model produce a non-linear curve for a regression line when we know that regression lines from a linear regression model can only be straight? stats.stackexchange.com/questions/487005/… $\endgroup$
    – develarist
    Sep 23, 2020 at 16:57
  • $\begingroup$ What basis expansion of the predictors (not the outcome) would allow you to model a multiplicative effect? $\endgroup$
    – Eoin
    Sep 23, 2020 at 17:00
  • 1
    $\begingroup$ @develarist you can perform linear regression against a nonlinear function of the explanatory variables. If the real, but unknown natural relationship is something like $y = a + b \cdot sin(x) + c \cdot cos(x^2)$, then you can perform linear regression on a formula such as that. $\endgroup$
    – abalter
    Sep 23, 2020 at 17:42
  • 2
    $\begingroup$ @Devel I explain this phenomenon, with illustrations, at stats.stackexchange.com/a/354256/919. The short answer is, when the regressor variables are non-linearly related, then a linear function of those variables typically looks like a nonlinear function of any proper subset of those variables. $\endgroup$
    – whuber ♦
    Sep 23, 2020 at 18:13
  • 1
    $\begingroup$ The ordinary least squares model doesn't care, in any way, whether the second regressor variable $x^2$ is a function of $x.$ Indeed, in old software one often had to process the data to create a separate column to hold the values of $x^2$ and then perform a multiple regression of the $y$ values against the $x$ values and the values in the second column. This little anecdote might help clarify the boundary between conducting a statistical procedure (such as OLS regression) and interpreting its output: only at the latter stage will you account for the relation between the regressors. $\endgroup$
    – whuber ♦
    Sep 23, 2020 at 19:38
0
$\begingroup$

I don't believe it's possible in general to do this with ordinary least squares, since OLS is at heart a trick to calculate $\hat \beta$ in $E[y] = \bf{X}\hat\beta$ using matrix division.

It can be done more generally, though.

I think the tricky bit is figuring out exactly what you mean by each predictor having an additive or multiplicative effect on the response. For example, with two predictors, do you mean:

$$ y = (\beta_0 \times \beta_2 x_2) + \beta_1 x_1 ?\\ y = (\beta_0 + \beta_1 x_1) \times \beta_2 x_2 ?\\ y = \beta_0 + (\beta_1 x_1 \times \beta_2 x_2) ? \\ $$

...and there's probably others as well. Of these, the first (multiplication before applying the additive effects) is the simplest to estimate, as it has fewer high-order multiplicative terms, and is more likely to correspond to the model you intended.

Unfortunately, even this isn't simple to estimate, since the predictions with $\beta_0 = 2, \beta_2 = 2$ → $y = (2 \times 2 \times x_2) + \beta_1 x_1$ are the same as those when
$\beta_0 = 1, \beta_2 = 4$ → $y = (1 \times 4 \times x_2) + \beta_1 x_1$.

The best way around this is to use a Bayesian estimation tool like Stan to set reasonable priors on your model parameters (for example that the multiplicative effect, $\beta_2$, should be close to 1), and find the best parameter estimates that are consistent with those priors.

Improve this answer
$\endgroup$
4
  • 2
    $\begingroup$ You are considering additive models with interactions. The usual meaning of a "multiplicative effect" is that the model satisfies a partial differential equation $\partial y/\partial x_i = \alpha_i y.$ $\endgroup$
    – whuber ♦
    Sep 23, 2020 at 17:02
  • $\begingroup$ I'm not familiar with this framing, thanks. Does the first model not satisfy this, since $\partial y/\partial x_2 = \beta_2 y$? $\endgroup$
    – Eoin
    Sep 23, 2020 at 17:11
  • 2
    $\begingroup$ In your first model $\partial y/\partial x_2 = \beta_0\beta_2$ is not a constant multiple of $y.$ In fact, all three of your models are standard regression models but have been overparameterized; none of them is a multiplicative effects model. $\endgroup$
    – whuber ♦
    Sep 23, 2020 at 17:17
  • 2
    $\begingroup$ Fair enough. I'll come back to this (and amend or remove my answer) tomorrow. $\endgroup$
    – Eoin
    Sep 23, 2020 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.