# How to test an interaction effect with a non-parametric test (e.g. a permutation test)?

I have two categorical / nominal variables. Each of them can take only two distinct values (so, I have 4 combinations in total).

Each combination of values comes with a set of numerical values. So, I have 4 sets of numbers. To make it more concrete, let us say that I have male / female and young / old as the nominal variables and I have weight as the dependent numerical "output".

I know that transition from male to female does change the average weight and these changes are statistically significant. So, I can calculate a gender factor. The same is applicable to the age variable. I do know that transition from young to old does change the average weight and I can calculate the corresponding age factor.

Now, what I really want to see if the data proves that transition from young-females to old-males is more that combination of gender- and age-factors. In other words, I want to know if data prove that there are "2D effects" or, in other words, that age- and gender-effects are not independent. For example, it might be that becoming old for males increase the weight by factor 1.3 and for female the corresponding factor is 1.1.

Of course I can calculate the two mentioned factors (age factor for males and age factor for females) and they are different. But I want to calculate the statistical significance of this difference. How real is this difference.

I would like to do a non-parametric test, if possible. Is it possible to do what I want to do by mixing the four sets, shuffling them, re-splitting and calculating something.

• One difficulty with dealing with interaction nonparametrically is that a monotonic transformation of the response can remove interaction that was present, induce interaction where it was absent, or flip the direction of interaction. This suggests that rank-based approaches, for example, may not do what you'd expect them to. May 28, 2016 at 2:12
• With permutation tests on the original variables, you don't have that problem but it turns out there are no exact tests for interaction. You can get some approximate tests. Apr 24, 2017 at 1:19

There are nonparametric tests for interaction. Roughly speaking, you replace the observed weights by their ranks and treat the resulting data set as heteroskedastic ANOVA. Look e.g. at "Nonparametric methods in factorial designs" by Brunner and Puri (2001).

However, the kind of nonparametric interaction you are interested in cannot be shown in this generality. You said:

In other words, I want to know if data prove that there are "2D effects" or, in other words, that age- and gender-effects are not independent. For example, it might be that becoming old for males increase the weight by factor 1.3 and for female the corresponding factor is 1.1.

The latter is impossible. Nonparametric interaction must involve a sign change, i.e. growing old increases males' weight but decreases females weight. Such a sign change remains even if you monotonically transform the weights. But you can choose a monotonous transformation on the data that maps the weight increase by factor 1.1 as close as you want to 1.3. Of course, you will never show a difference to be significant if it can be as close as you want.

If you really are interested in interactions without sign change, you should stick to usual parametric analysis. There, monotonous transformations that "swallow the difference" aren't allowed. Of course, this is again something to keep in mind by modeling and interpreting your statistics.

As others have noted, this can be modeled linearly with an interaction. You're interacting two dummies, and there's nothing non-linear about this. Given the model: $$wt = \alpha +b_1age+b_2gender+b_3age*gender+\epsilon$$ The 'gender' marginal effect is the partial derivative:

$$\frac{\partial wt}{\partial gender} = b_2 + b_3age$$

See how if gender and age can only take values of 0 or 1, we're essentially only looking at a difference in means for four different groups? That is, we only have the four different combinations we can plug into the above equations: (1) $gender = 0$ and $age=0$, (2) $gender = 1$ and $age = 1$, (3) $gender = 0$ and $age = 1$, and (4) $gender = 1$ and $age = 0$. Thus, your specific example is equivalent to a comparison between four group means.

It might also be helpful to see this discussion to understand how the above is equivalent to ANOVA with two interacted nominal variables. As another way to restate the fact that with your specific example, (again, because there are only four possible combinations of age and gender) we could also specify a model like the following, without an explicit interaction term: $$wt = \alpha + b_1young.male + b_2old.male + b_3young.female + \epsilon$$

Where $old.female$ is omitted as your reference category, and for example, the coefficient $b_1$ will be a difference in means between $old.female$ and $young.male$. Where the intercept $\alpha$ will also be equal to average $wt$ within $old.female$ (again, the reference category).

Try it out with your own data. With a linear model with an interaction, an ANOVA with an interaction, or using dummies for each of the groups with no interaction, you'll get the same results. Pretty cool, huh? A statistics book might discuss each of these methods in a diferent chapter$\dots$ but all roads lead to Rome. Really, seeing how this works with your own data is one of the best ways to learn.

The above examples are thus an overly complicated way to get at this conclusion (that we're really just comparing four group means), but for learning about how interactions work, I think this is a helpful excercise. There are other very good posts on CV about interacting a continuous variable with a nominal variable, or interacting two continutous variables. Even though your question has been edited to specify non-parametric tests, I think it's helpful to think through your problem from a more conventional (i.e., parametric) approach, because most non-parametric approaches to hypothesis testing have the same logic but generally with fewer assumptions about specific distributions.

But the question asked specifically for a non-parametric approach, which might be more appropriate, for example, if we didn't want to make certain assumptions about the normality of $wt$. An appropriate non-parametric test would be Dunn's test. This test is similar to the Wilcoxon-Mann-Whitney rank-sum test but with more than two categories.

Other permutation tests might also be appropriate if you had a specific difference in means you were testing, for example, $old.men$ vs. $young.women$. Whether or not you use R, the 'coin' package documenation provides a good summary of different non-parametric tests, and under what circumstances these tests might be appropriate.

### Short aside on "significant" interactions

Sometimes, you'll see statments like, "the interaction between $x_1$ and $x_2$ was statistically significant." Such statements are not necessarily wrong, but they are misleading. Usually, when an author writes this, they are saying that the coefficient on the interaction term was statistically signficant. But this is an unconditional effect in a conditional model. A more accurate report would say that "$x_1$ was statistically signficant over 'some values' of $x_2$," where all other covariates were held constant at some reasonable value, like a mean, median or mode. But once more, if we only have two covariates that can only take values of 0 or 1, that means that we're essentially looking at four group means.

### Worked Example

Let's compare results from the interaction model with results from Dunn's test. First, let's generate some data where (a) men weigh more than women, (b) younger men weight less than older men, and (c) there is no difference between younger and older women.

set.seed(405)
old.men<-rnorm(50,mean=80,sd=15)
young.men<-rnorm(50,mean=70,sd=15)
young.women<-rnorm(50,mean=60,sd=15)
old.women<-rnorm(50,mean=60,sd=15)
cat<-rep(1:4, c(50,50,50,50))
gender<-rep(1:2, c(100,100))
age<-c(rep(1,50),rep(2,100),rep(1,50))
wt<-c(old.men,young.men,young.women,old.women)
data<-data.frame(cbind(wt,cat,age,gender))
data$cat<-factor(data$cat,labels=c("old.men","young.men","young.women","old.women"))
data$age<-factor(data$age,labels=c("old","young"))
data$gender<-factor(data$gender,labels=c("male","female"))


Estimate the interaction model and get predicted $wt$ from marginal effect (w/ 'effects' package). See here for why we don't want to interpret the unconditional effects in a model like this. Instead, we want to interpret marginal effects. The model does a decent job of detecting the differences we imposed when we generated our example data.

mod<-lm(wt~age*gender,data)
library(effects)
allEffects(mod)

model: wt ~ age * gender

age*gender effect
gender
age         male   female
old   80.61897 57.70635
young 67.78351 56.01228


Need to calculate a standard error or confidence interval for your marginal effect? The 'effects' package referenced above can do this for you, but better yet, Aiken and West (1991) give you the formulas, even for much more complicated interaction models. Their tables are conveniently printed here, along with very good commentary by Matt Golder.

Now to implement Dunn's test.

#install.packages("dunn.test")
dunn.test(data$wt, data$cat, method="bh")

Kruskal-Wallis chi-squared = 65.9549, df = 3, p-value = 0

Comparison of x by group
(Benjamini-Hochberg)
Col Mean-|
Row Mean |    old.men   young.me   young.wo
---------+---------------------------------
young.me |   3.662802
|    0.0002*
|
young.wo |   7.185657   3.522855
|    0.0000*    0.0003*
|
old.wome |   6.705346   3.042544  -0.480310
|    0.0000*    0.0014*     0.3155


The p-value on the Kruskal-Wallis chi-squared test result suggests that at least one of our groups 'comes from a different population.' For the group-by-group comparisons, the top number is Dunn's z-test statistic, and the bottom number is a p-value, which has been adjusted for multiple comparisons. As our example data were rather artificial, it's unsurprising that we have so many small p-values. But note the bottom-right comparison between younger and older women. The test correctly supports the null hypothesis that there is no difference between these two groups.

So, both the interaction model and Dunn's test lead us to similar conclusions. In all of the examples given above, we are somehow comparing group means. And, while there are certainly more strait-forward approaches to comparing group means, I have tried to illustrate how comparing group means can also be understood as an interaction or "2D effect," with some model specifications, specifically with nominal interactions. I think that understanding this is helpful for understanding more complicated models with interaction effects. I'm going to link to this artice once more, just because I think it should be required reading for anyone working with interactions (there's a reason this article has been cited over 3k times$\dots$).

UPDATE: Given other answers, this answer has been updated to dispute the idea that this requires any form of non-linear modeling, or that -- given OP's specific example of two binary covariates, i.e., four groups -- that there must be a sign change to asesses this non-parametrically. If age were continuous, for instance, there would be other ways to approach this problem, but that was not the example given by OP.

• You don't use the structure of two crossed factors. You merely compare four groups. Dunn's test is not about interaction at all. May 26, 2016 at 15:55
• Agreed, Dunn's test is not about interaction. However, the question asks specifically about an interaction between two binary variables. My answer demonstrates how this is equivalent to comparing the four groups. If interaction terms are new to OP, hopefully this is a helpful illustration. May 26, 2016 at 17:07

If you believe that the effects of age and gender are more than just the individual effects, you may consider the model $weight_i = \alpha \cdot age_i + \beta \cdot gender_i + \gamma \cdot (gender_i\cdot age_i).$ The $\gamma$ coefficient captures the size of the "2D" effect of age and gender. You can check the t-statistic of $\gamma$ to get a rough idea on whether the $\gamma$ you observe in your model is significantly different from $\gamma = 0$.

Here's a very rough graphical example to show what this additional multiplicative term $gender_i\cdot age_i$ does.

In the model $response = x_1 + x_2$, we essentially try to fit a simple hyperplane to the data This is model is linear in the covariates, hence the linear shape you see in the plot above.

On the other hand, the model $response = x_1 + x_2 + x_1\cdot x_2$ is non-linear in $x_1$ and $x_2$ and hence allows for some level of curvature Failing to reject the hypothesis that $\gamma = 0$ is like failing to reject that there is some curvature of this form in the model.

In terms of a non-parametric test, you can do something along the lines of what you suggested by obtaining bootstrap standard errors for $\gamma$. This means that, several times you: 1) sample your data with replacement, 2) recalculate the linear mode, 3) get an estimate $\hat{\gamma}$. After you have many estimates of $\hat{\gamma}$, you can use the $50 \pm p\%$ quantile to set up a non-parametric $2p\%$ confidence interval for $\gamma$. For more on this, google "bootstrap standard errors".

• How can this be non-linear if x1 and x2 can only take values of 0 or 1? How would gamma in your example explain any form of curvature? May 23, 2016 at 12:10
• It doesn't matter what the domain is, it's still non-linear because the function cannot be written as a linear combination of its arguments (i.e. $\nexists \alpha \in \mathbb {R}^2: x_1 + x_2 + x_1 x_2 = \sum_{i=1}^2 \alpha_i x_i$). For your second point, notice I carefully said "a very rough graphical example." This is a continuous analogue of the binary case. May 23, 2016 at 17:00
• I will add, however, that when the domain is binary (which is like.the vertices of the 2D cube), you can treat this function linearly. But the functional form is strictly non-linear. May 23, 2016 at 17:12
• @MustafaMEisa, I've never seen an interaction term in a linear model explained in terms of "the vertices of a 2D cube." It would be informative if you could elaborate. May 27, 2016 at 20:07
• @HorstGrünbusch, I'm also curious as to your comment on this answer, as you have already given a helpful comment on my answer. May 27, 2016 at 20:08

So you have these random variables:

• $A$ - takes values in $\mathbb{N}$. This is what we call age.
• $S$ - takes values in $\{\text{male},\text{female}\}$. This is what we call sex.
• $W$ - takes values in $]0, \infty[$. This is what we call weight.

And you have these probability mass/density functions:

• $f_W$ - density of r.v. $W$.
• $f_{W,A}$ - joint density of r.v. $W,A$.
• $f_{W,S}$ - joint density of r.v. of $W,S$.
• $f_{W,A,S}$ - joint density of r.v. $W,A,S$.

You know that there exists weight $w$, age $a$ and sex $s$ such that:

• $f_{W,A}(w,a) \ne f_W(w)$, therefore weight is not independent of age.
• $f_{W,S}(w,s) \ne f_W(w)$, therefore weight is not independent of sex.

Now, you wish to find out whether age and sex are independent as they are jointly/combinatorially related to weight. In other words, you wish to find this: $$f_{W,A,S}(w,a,s) \ne f_{W,A}(w,a) \ne f_{W,S}(w,s)$$

If you show that there exists weight $w$, age $a$ and sex $s$ that satisfy the above, then you will show that age and sex have a combinatoric effect on weight.

However, you don't know the true joint PDFs above. Since you want to limit yourself to non-parametric methods, your task now is to find these non-parametric estimations:

• $\hat f_{W,A}(w,a)$.
• $\hat f_{W,S}(w,s)$.
• $\hat f_{W,A,S}(w,a,s)$.

And then show that:

• Your density estimations are accurate enough.
• The probability that $\hat f_{W,A,S}(w,a,s) \ne \hat f_{W,A}(w,a) \ne \hat f_{W,S}(w,s)$ is very high.
• Or that the probability that $\hat f_{W,A,S}(w,a,s) = \hat f_{W,A}(w,a) = \hat f_{W,S}(w,s)$ is very low.

That would be checking for interaction effects. Linear Modeling would be able to check such thing but it is not non-parametric so I guess another tool must be used.

How are you checking your age and gendereffect up until now ?