# Which ML algorithm can learn non-linear interaction effects?

In my regression problem I have numeric input columns "A", "B" and "C" and the numeric target "Target".

The relationship is: The higher "C", the more impact has "B" - the lower "C" the more impact has "A" in order to predict "Target". Now that relationship is non-linear, but more like quadratic.

Which regression models are in theory able to learn that kind of interaction relationships without manually adding interaction terms?

• If you know what is the interaction, what is the problem with adding it manually?
– Tim
Commented Oct 10, 2020 at 10:47
• I am looking for a generic approach that I can apply on different problems with a similar structure but different relationships. Commented Oct 10, 2020 at 10:52
• Are you open to tree-based models, like random forests and/or gradient boosting? Commented Oct 12, 2020 at 23:00
• To my understanding tree-based models like this are not able to learn that kind of interaction functions but just partition the data Commented Oct 13, 2020 at 6:17
• @HansHupe on the contrary, trees can learn interactions. See stats.stackexchange.com/questions/147594/… Commented Oct 16, 2020 at 16:20

Any universal approximators can do it. You need a term like $$A(\beta_A+\beta_{A\times C}\times C)$$ to appear, so the interaction between $$A$$ and $$C$$ suffices.

$$A\times C = \frac{(A+C)^2-A^2-C^2}{2}$$

If you have an universal approximator, it can (locally) approximate the quadratic form somewhere in its formulation, giving you the interaction without explicitly multiplying $$A$$ and $$C$$.

Then, the only thing that matters is selecting a universal approximator. Neural Networks are in general universal approximator, and so are kernel machines with infinite dimensional kernel spaces (like the radial basis function, for example) too.

On neural networks, if you have as inputs $$A,B,C$$, then with two hidden layers and the square as the activation function you already achieves the possibility of interactions.

Consider the column vector $$x = [A, B, C]$$:

$$\hat y = W_2\sigma (W_1 x+b_1)+b_ 2$$

$$W_1 x$$ passes weighted sums of the initial features, $$h_1 = \sigma(W_1 x+b_1)$$ square them and finally $$W_2h_1+b_ 2$$ makes weighted sums of the squared items.

MARS (Multivariate Adaptive Regression Splines) are able to detect automatically non-linear interactions between explanatory variables without manually adding them in the model

Perhaps you can add polynomial interaction terms with some high orders and use lasso regression? You can get some clue from the coefficients of these terms. That being said, ML algorithms are usually for prediction instead of estimating the effects.

• Adding manually different orders of interaction terms require domain knowledge, which I want to minimize, also there is a risk of overfitting when interaction terms are added between all variables. I am kind of disappointed that ML algorithms are not able to learn that effects. At the moment I am experimenting with MARS. Commented Oct 11, 2020 at 8:41

If you need explicit and interpretable interactions you should use MARS of 2nd or 3rd degree. If you need explicit but not interpretable interactions (you will not be able to extract the interaction features after fitting the model) you could use SVM with polynomial kernels. If you are ok with implicit and flexible interactions, as Firebug said, you can use a universal approximator such as a Neural Network with non-linear activations. I guess you could also use SVM with radial basis kernel for this purpose, as it is also a universal approximator, however, I am not completely sure of how would be this model able to model interactions (I posted a question specifically for this matter that has not been answered yet).