A lot of applications in statistics and machine learning model a phenomenon by second order interactions of variables and get good results. By second order interactions I mean, for a general variable $x = [x_1, \cdots, x_n]$, a model like $\sum_{i=1}^n \alpha_i x_i + \sum_{i=1}^n \sum_{j=1}^n \alpha_{ij} x_{i}x_{j} $.

My question is how powerful are these models compared to linear models? Can we characterize this mathematically?

I know one simple reason is that such models can represent non-linear functions and going to higher-order models will give us more flexibility but less tractability. Does quadratic models have general edge over higher models just because of tractability? Are there other reasons at play here? What applications require models with higher-order(>2) interactions?

Please excuse me if the question is too general for the site.


The higher order interaction model contains all lower order interaction models and is thus "more generally applicable".

Note that such a model is still linear in the unknown parameters $\alpha_{i}, \alpha_{i,j}$ and thus the whole mechanism of linear models still applies. Modeling higher order interactions is in this sense not more or less powerful than modeling no interactions as it still falls into the notion of a linear model. No new mathematics is necessary to fit a first order interaction mode if one can fit a linear model that is also linear in the variables.

You should also consider the data that you are analyzing. Using higher order models will introduce more parameters which might lead to overfitting if you have a small sample size. On the other hand modeling too few interactions may result in you missing interesting effect in your data. Ultimately what you may be looking for is model selection.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.