How to interpret the relation or interaction between two variables or features in general for classification?

Question

Suppose I have several features like $X_1, X_2, X_3$ etc. In my model, I have to know whether $X_1$ and $X_2$ will have an impact together. I read somewhere we can make a new feature by $( X_1*X_2 )$, which will have a new coefficient, lets say $C$. My question is, in R, for regression we can use a default function to combine two variables? But, what's happening inside that function? Are they doing $X_1*X_2$ or $X_1+X_2$? In python sklearn, as of my knowledge there is no way of doing that.

So, please would someone help me and explain the relevance of combining two features theoretically and practically?

Answer 1

What you are talking about are basic inteaction effets. You could think forexample of simple model:

y = a + b1*x1 + b2*x2 + e

If we add and interaction-term to the model depicted, the following term consequences.

y = a + b1*x1 + b2*x2 + b3*x1*x2 + e

By simply reordering, we find

y = (a + b1*x1) + x2*(b2+b3*x1) + e

The first parenthesis depicts the intercept and the second the regression’s slope. However, both elements are subject to the level of x1. Despite term appearing rather econometrically tame, Braumöller (2004) shows that interpreting models with multiplicative terms often goes askew especially when turning to the lower-order coefficients b1 and b2.

The regression’s slope will equal b1 only if x2 equals zero. Thus b1 denotes the simple effect of x1 on y. Similar, x2 denotes the simple effect on y since y = a + b2*x2 + e ; for x1=0. Similarly, a is the intercept when both x1 anx2 are zero.

b3 finally depicts the interactions between x1and x2. And here lies the crux: First, you can only interpret x3if you incorporate both x1and x2in your regression. For a detailed analysis of this matter see Whisman, Mark A., and Gary H. McClelland. "Designing, testing, and interpreting interactions and moderator effects in family research." Journal of Family Psychology 19.1 (2005): 111. Second, you need "meaniful" values for x1=0and x2=0. One idea might be a centering at meaningful values or at means. For a more detailed discussion of this matter check here: When conducting multiple regression, when should you center your predictor variables & when should you standardize them?

Statistic packages like Stata and R usually do everything for you. Practically, you can incorporate x1, x2 and a third variable x1*x2. But usually it is better to merely incorporate x1 and x2 and "tell" R/Stata to gauge an interactio effect too ( x1##x2 or x1#x2 in Stata). R is a little different. Check here: Different ways to write interaction terms in lm?