# How to interpret the relation or interaction between two variables or features in general for classification?

Suppose I have several features like $X_1, X_2, X_3$ etc. In my model, I have to know whether $X_1$ and $X_2$ will have an impact together. I read somewhere we can make a new feature by $( X_1*X_2 )$, which will have a new coefficient, lets say $C$. My question is, in R, for regression we can use a default function to combine two variables? But, what's happening inside that function? Are they doing $X_1*X_2$ or $X_1+X_2$? In python sklearn, as of my knowledge there is no way of doing that.

So, please would someone help me and explain the relevance of combining two features theoretically and practically?

What you are talking about are basic inteaction effets. You could think forexample of simple model:

y = a + b1*x1 + b2*x2 + e

If we add and interaction-term to the model depicted, the following term consequences.

y = a + b1*x1 + b2*x2 + b3*x1*x2 + e

By simply reordering, we find

y = (a + b1*x1) + x2*(b2+b3*x1) + e

The first parenthesis depicts the intercept and the second the regression’s slope. However, both elements are subject to the level of x1. Despite term appearing rather econometrically tame, Braumöller (2004) shows that interpreting models with multiplicative terms often goes askew especially when turning to the lower-order coefficients b1 and b2.

The regression’s slope will equal b1 only if x2 equals zero. Thus b1 denotes the simple effect of x1 on y. Similar, x2 denotes the simple effect on y since y = a + b2*x2 + e ; for x1=0. Similarly, a is the intercept when both x1 anx2 are zero.

b3 finally depicts the interactions between x1and x2. And here lies the crux: First, you can only interpret x3if you incorporate both x1and x2in your regression. For a detailed analysis of this matter see Whisman, Mark A., and Gary H. McClelland. "Designing, testing, and interpreting interactions and moderator effects in family research." Journal of Family Psychology 19.1 (2005): 111. Second, you need "meaniful" values for x1=0and x2=0. One idea might be a centering at meaningful values or at means. For a more detailed discussion of this matter check here: When conducting multiple regression, when should you center your predictor variables & when should you standardize them?

Statistic packages like Stata and R usually do everything for you. Practically, you can incorporate x1, x2 and a third variable x1*x2. But usually it is better to merely incorporate x1 and x2 and "tell" R/Stata to gauge an interactio effect too ( x1##x2 or x1#x2 in Stata). R is a little different. Check here: Different ways to write interaction terms in lm?

• So helpful . I will read the links. So, the effect of above mentioned things will remain the same or atleast having same interpretation in classification also. – Sarath R Nair Jun 9 '15 at 13:17
• That is true - but only if you leave x1 and x2 in the regression when adding the interaction. Only having the interaction without x1 and x2 leads to false interpretation . If the above answers your question, try to indicate so via the button :) – Rachel Jun 9 '15 at 13:21
• Sleep - Actually, i didn't understand the last comment ? Can you please be specific when you are free . – Sarath R Nair Jun 10 '15 at 4:07
• Sure, no problem. I merely wanted to point out that you need to include both the individual effects and the interaction term if you want to be able to have meaningfull interpretations. If you were to add an interaction and not the individual effects, a meaningful interpretation is hard (or even impossible). That is why packages like Stata/R/SAS and alike allow a notation that includes the individual and the interaction in on step. For Stata you might use x1##x2. this results in x1, x2 and x1*x2 in one step. – Rachel Jun 10 '15 at 6:26
• My pleasure! Also, if you categorical data, make sure you think about how to code it (e.g. dummy 0/1 vs effect -1/1 and alike). – Rachel Jun 11 '15 at 6:10