# How powerful are second order interactions?

A lot of applications in statistics and machine learning model a phenomenon by second order interactions of variables and get good results. By second order interactions I mean, for a general variable $$x = [x_1, \cdots, x_n]$$, a model like $$\sum_{i=1}^n \alpha_i x_i + \sum_{i=1}^n \sum_{j=1}^n \alpha_{ij} x_{i}x_{j}$$.

My question is how powerful are these models compared to linear models? Can we characterize this mathematically?

I know one simple reason is that such models can represent non-linear functions and going to higher-order models will give us more flexibility but less tractability. Does quadratic models have general edge over higher models just because of tractability? Are there other reasons at play here? What applications require models with higher-order(>2) interactions?

Please excuse me if the question is too general for the site.

Note that such a model is still linear in the unknown parameters $$\alpha_{i}, \alpha_{i,j}$$ and thus the whole mechanism of linear models still applies. Modeling higher order interactions is in this sense not more or less powerful than modeling no interactions as it still falls into the notion of a linear model. No new mathematics is necessary to fit a first order interaction mode if one can fit a linear model that is also linear in the variables.