5
$\begingroup$

we use models with multiplicative interaction effects when relationship between independent variable and dependent variable are non-additive.

My question is, Are all models with multiplicative interaction effects non-linear? and all models with additive interaction effects linear?

Also, With non-linearity, the effect of independent variable on dependent variable depends on the value of independent variable, in effect, independent variable somehow interacts with itself. Does that mean that an independent variable(x1) interacts with itself(x1)? or does it mean that independent variables(x1) interact with other independent variables(x2, x3...) and not with itself?

I am confused with concepts of of linearity, non-linearity, additivity and non-additivity (multiplicativity).

Any help is much appreciated.

$\endgroup$

2 Answers 2

9
$\begingroup$

we use models with multiplicative interaction effects when relationship between independent variable and dependent variable are non-additive.

My question is, Are all models with multiplicative interaction effects non-linear? and all models with additive interaction effects linear?

The answer to such a question depends on what you mean when you say 'linear' and 'nonlinear', and what domain of models you're restricting yourself to.

Usually the terms 'linear' and 'nonlinear' in statistical models refers to linearity in the parameters, not the variables.

So for example, $y = \alpha x^2 +\epsilon$ is linear in $\alpha$ though not in $x$, while $y = \exp(-\alpha) x +\epsilon$ is non-linear in $\alpha$, though it is in $x$. In usual parlance, the first is a linear model and the second is not. However, in those cases at least both may be turned into models that are linear in both the parameters and the predictors - in the first case by the transformation $x^* = x^2$, giving a model that has a linear relationship between $y$ and $x^*$, and in the second case by the reparameterization $\alpha^* = \exp(-\alpha)$.

As such a standard general linear model (regression-type model) with multiplicative interaction is linear in the parameters, even though it's not linear in either predictor (IV). However, note that even in terms of the IVs, it is conditionally linear - fix one of the IVs and the relationship is linear in the other.

[Minor mathematical aside: It should be noted that when we're taking about the relationships of $y$ and some $x$ being linear (in this sense rather than the 'makes a straight-line' sense), if we recognize we're using homogeneous co-ordinates in regression, it is linear. I mention it because I have seen people with enough mathematical background to be familiar with the mathematical definition of linearity object that 'linear regression is not linear'.]

all models with additive interaction effects linear?

If I understand what you're even asking with 'additive interaction effects', there's really no such thing. If it's additive it's already in the main effects and there's nothing left over for some notional 'interaction'.

Also, With non-linearity, the effect of independent variable on dependent variable depends on the value of independent variable,

Only if you think of 'effect' as inherently linear

in effect, independent variable somehow interacts with itself.

This way lies much confusion. Why not just think of there being a relationship that's described by some curve rather than by a straight line?

--

Edit to address followup questions:

What do you mean when you say "what domain of models you're restricting yourself to"?

When you said "all models with multiplicative interaction effects" you presumably meant 'all models' in some class, such as regression models, or general linear models, or generalized linear models, or ... the list could go on for some time.

Thanks! for noting about linearity. For the longest time, even I thought being linear meant the relationship was a straight line.

Me too.

This does clear some doubts, but raises a few questions. So, if we recognize we're using homogeneous co-ordinates in regression, it is linear.

In terms of $x$'s - the actual columns of the $X$-matrix in a regression - it's linear in that linked mathematical sense if you realize you're working with homogeneous co-ordinates.

A multiple linear regression is already linear in the mean parameters (i.e. the $\beta$ vector, the parameters other than $\sigma^2$), without any such need to invoke homogeneous co-ordinates. That I was referring to the relationship with the $x$'s when I raised homogeneous co-ordinates was explicitly stated.

Also, Did you mean to say "Only if you think of 'effect' as inherently nonlinear instead of linear?

Nope. The way you phrased the question I was responding to only makes sense if you take the word 'effect' to imply linearity, otherwise the whole notion of 'interaction with itself' seems to be utterly meaningless. How is one to interpret the phrase?

What I meant to ask was. I read somewhere that "with non-linearity, the effect of X on Y depends on the value of X and X somehow interacts with itself".

I regard the statement as an unhelpful attempt at analogy, and, as already explained, I think you should not think about it this way. Not everything that someone writes down is useful.

Does this mean that X interacts with itself(X)? or does it mean that X interact with other variables(X,W etc.) if any?

I'm not going to make any further attempt to interpret something that doesn't really make sense as a bare, general, statement without first having more clarification of its intent. I've suggested a way to interpret it that makes at least a little sense. If you want to interpret it more generally, explaining what it means would be up to you - or the original author of it.

I expect if you were to ask "what, exactly, does it mean*, you would receive an answer that contained a number of hidden premises, and one of those premises would rely, directly or indirectly, on taking underlying meaning of 'effects' to be linear, when we have no good reason to do that.

$\endgroup$
6
  • $\begingroup$ What do you mean when you say "what domain of models you're restricting yourself to"? Thanks! for noting about linearity. For the longest time, even I thought being linear meant the relationship was a straight line. This does clear some doubts, but raises a few questions. So, if we recognize we're using homogeneous co-ordinates in regression, it is linear. Is this true for parameters, third, fourth, fifth variables? or just X(IV)? $\endgroup$
    – user793468
    Mar 19, 2013 at 23:27
  • $\begingroup$ Also, Did you mean to say "Only if you think of 'effect' as inherently nonlinear instead of linear? What I meant to ask was. I read somewhere that "with non-linearity, the effect of X on Y depends on the value of X and X somehow interacts with itself". Does this mean that X interacts with itself(X)? or does it mean that X interact with other variables(X,W etc.) if any? $\endgroup$
    – user793468
    Mar 19, 2013 at 23:28
  • $\begingroup$ See my edited answer $\endgroup$
    – Glen_b
    Mar 19, 2013 at 23:46
  • $\begingroup$ I understand your point on linear relationship that when we're taking about the relationships of y and some x being linear, if we recognize we're using homogeneous co-ordinates in regression, it is linear and that the curve plot does not matter. But wouldn't homogeneous co-ordinates always plot a straight line? $\endgroup$
    – user793468
    Mar 20, 2013 at 21:41
  • $\begingroup$ If the equation is linear in a single $x$, then an equation in homogenous co-ordinates will generate a straight line, yes. The original point was just an aside about what 'linear' means to different people - because there's at least three ways to interpret it (linear in parameters, linear in x's under homogeneous coordinates, and generating a straight line for the plot of $y$ vs $x$). All three apply. $\endgroup$
    – Glen_b
    Mar 20, 2013 at 21:46
0
$\begingroup$

To understand this, you must first be clear about the disctinction between features and parameters.
Features - this is basically the input columns of your training data.
Parameters - these are basically the weights of your model that you want to learn
For example, for a linear regression model with two features, you make the assumption that the response depends on the features through the following relationship: $$Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \epsilon$$ The features are $X_1$ and $X_2$ whereas $\beta_0, \beta_1, \beta_2$ are the parameters

Linearity

Linearity is meant in terms of the parameters of the model. So if the response(or a function of the response) can be written as being linear in the parameters, we say the model is linear.
Eg. linear regression with two features shown above
polynomial regression - $Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \beta_3 X_1^2+ \beta_4 X_2^2 + \beta_5 X_1 X_2 + \epsilon$
logistic regression - $log (\frac{p_i}{1-p_i}) = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \epsilon$

Additivity

Additivity is meant in terms of the features of the model. So if the response(or a function of the response) can be written as being additive in the features (or a function of features), we say the model is additive. Eg. linear regression with two features shown above
logistic regression - $log (\frac{p_i}{1-p_i}) = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \epsilon$
because $g(Y) = f(X_1) + g(X_2) + \epsilon$

Models can be (linear & additive), (linear & not additive), (nonlinear & additive), (nonlinear & not additive)
The following models are linear & not additive:
polynomial regression - $Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \beta_3 X_1^2+ \beta_4 X_2^2 + \beta_5 X_1 X_2 + \epsilon$
because $X_1, X_2$ terms cannot be split into separate additive functions of $X_1$ and $X_2$

The following models are nonlinear & additive:
$Y = exp(\beta_1 X_1) + \beta_2 X_2 + \epsilon$
since this is not linear in the parameters

The following models are nonlinear & not additive:
$Y = exp(\beta_1 X_1) + \beta_2 X_2 + \beta_3 X_1 X_2 + \epsilon$
neither linear in parameters nor can it be split into separate additive functions of $X_1$ and $X_2$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.