I'm looking for the best way to test the hypothesis that two groups are different considering the relationship between two variables.

There is a population of individuals and, for each individual $x$, I know a value $m(x)$ from 0 to 100 and a binary value $s(x)$. I know there is a strong correlation between $m$ and $s$. (Higher $m$ means $s$ is more likely to be 1.) Each individual also is a member of one of a small finite set of groups, $g(x)$.

By binning on $m$ and plotting bin number versus the percentage of $s$s that are 1, separately for each group, I can see that there appear to be differences: E.g. those in group $A$ with low $m$ scores are more likely to have $s=1$ than those with low $m$ scores who are not in group $A$. I can also see that members of group $B$ seem to have a lower correlation between $m$ and $s$ than members of other groups. But is this observed difference significant?

So, for each group $A$ I'd like to test the hypothesis that being in group $A$ means that the relationship between $m$ and $s$ is significantly different than for those not in $A$. The null hypothesis is that members of group $A$ are just the same as those who are not in group $A$.

My question then is: what sort of analysis should I look into applying in order to test this sort of hypothesis?


1 Answer 1


To me this looks like the perfect setup for logistic regression because the variable $s(x)$ is 0 or 1 and $m(x)$ is numeric.

Briefly, you can assume that the probability that $s(x) = 1$ follows the model below saying that the relationship between $s(x)$ and $m(x)$ is monotonic and S-shaped (more precisely logistic).

$$P(s(x) = 1) = \frac{1}{1+e^{\beta_0 + \beta_1 m(x)}}.$$

You can make the model more complex and add a variable $g(x)$, say, which is 1 when the element $x$ belongs to group A. The model becomes the following.

$$P(s(x) = 1) = \frac{1}{1+e^{\beta_0 + \beta_1 m(x) + \beta_2 g(x)}}.$$

The null hypothesis in your case can be reformulated as $\beta_2 = 0$. If you are not familiar with linear models, this may be a little technical. Fortunately you do not need to be an ace to use the native R library. Below I show an example with fake data that illustrates how to use it.

# Create some (reproducible) random fake data.
m <- as.integer(runif(100, min=0, max=100))
beta_0 <- -4.0
beta_1 <- 0.1
s <- 1 / (1+exp(-(beta_0+beta_1*m)))
plot(m, s) # Enjoy nice S-shape.
base_model <- glm(s~m, family="binomial") # Issues a warning.
base_model$coefficients # beta_0 and beta_1 are estimated accurately.

Now we add an uninformative grouping function $g(x)$ and perform the test.

g <- runif(100) < 0.3
# Add g to the model.
model <- glm(s~m+g, family="binomial") # Issues a warning.

The last command outputs the following.

              Estimate Std. Error z value Pr(>|z|)    
(Intercept) -4.000e+00  8.570e-01  -4.668 3.05e-06 ***
m            1.000e-01  1.949e-02   5.132 2.87e-07 ***
gTRUE        1.077e-12  6.924e-01   0.000        1

See that the last line shows a z-value equal to 0 and a p-value equal to 1, which means that the null hypothesis is not rejected. With real data, this is never so clear, but you can replace s, m and g with your data and obtain the answer to your question in the same way.

Following the comments of @MichaelMayer I will try to discuss whether a more complex model is required, more particularly whether it should have an interaction term of the form $\beta_3 m(x) g(x)$. Here are the reasons that were on my mind at the time of writing the answer above.

  1. It seemed to me that the poster was not familiar with logistic regression so I opted for a simple but operative answer to give him a place to start.
  2. Hands on experience shows that if the relationship between $m$ and $s$ is different among groups, $\beta_2$ will often be significantly different from 0, even if this model is mis-specified.
  3. Without further information about the dataset and the kind of difference the poster is looking for, I opted for the most coarse-grained model.

If $\beta_2 \neq 0$, the S-shaped curve will be shifted in one of the groups (the probability that $m$ is 1 goes uniformly higher or lower). If $\beta_3 \neq 0$, the most pronounced effect is that the S will get flatter or steeper in one of the groups. An example will explain better. The R lines below create an ideal dataset with such an interaction term.

beta_0 <- -4
beta_1 <- 0.1
beta_3 <- 0.1
s <- 1 / (1+exp(-(beta_0+beta_1*m+beta_3*m*g)))
plot(m, s, col=1+g) # The points of group A in red

For this dataset, $\beta_2=0$ (it is not even used in the specification). Let us see what happens when testing for its nullity. We do it exactly as above.

model <- glm(s~m+g, family="binomial") # Issues a warning

This time we obtain the following.

            Estimate Std. Error z value Pr(>|z|)    
(Intercept) -4.32243    0.96382  -4.485  7.3e-06 ***
m            0.10816    0.02242   4.825  1.4e-06 ***
gTRUE        2.44032    0.89081   2.739  0.00615 ** 

The hypothesis $\beta_2=0$ is rejected. Of course, this is the wrong conclusion because $\beta_2$ is 0 by construction but... wait a minute, the groups are different! So even if we got the statistical modeling framework completely wrong, we got to the right conclusion about the groups. If there is a difference between the groups, it will often “leak” into the test for $\beta_2$ even if the relationship is not the right one (the model with $\beta_2$ is mis-sepcified, but still has predictive power).

Don’t get me wrong, there is no guarantee that this will happen, and I am not saying that interaction terms are useless in this context. If you are really interested in the flatness of the S-curve, you should incorporate the interaction term and specifically test its nullity. But from the context of the question, I gathered that the focus is really on the groups and whether or not they are different. I thought that this simple approach would already bring you a long way.

  • 1
    $\begingroup$ I wonder how a model without interactions may answer the OPs question. $\endgroup$
    – Michael M
    Jul 17, 2014 at 21:54
  • $\begingroup$ @MichaelMayer From the OP: "I'd like to test the hypothesis that being in group A means that the relationship between m and s is significantly different than for those not in A". Can you expand on your comment and explain why interaction terms are required? Why do you think that beta_2 = 0 does not imply that the relationships between m and s are identical inside and outside of group A? $\endgroup$
    – gui11aume
    Jul 17, 2014 at 22:24
  • $\begingroup$ The original question reformulated: Is the effect of m on (log odds of) s homogenous across dichotomized g? You can't answer this with an additive model. It is only about the interaction between g and m. Furthermore, there is the problem of multiple testing. $\endgroup$
    – Michael M
    Jul 18, 2014 at 6:48
  • $\begingroup$ @MichaelMayer Can you please answer the question? Why do you think that beta_2 = 0 does not imply that the relationships between m and s are identical inside and outside of group A? $\endgroup$
    – gui11aume
    Jul 18, 2014 at 7:26
  • 1
    $\begingroup$ @MichealMayer An answer from you would be greatly appreciated. It seems that some clarification of the question would be useful, but I'm not sure I understand the issues well enough at present to give that clarification. Is there a question I can answer to make that clarification? $\endgroup$ Jul 19, 2014 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.