I have run an experiment in which I'm comparing the behavior of two groups of unequal size (group1 and group2), where each subject in the group has to take one two possible actions. So I'm ending up with data of the form:

        action_A    action_B
group1     3320       2273
group2    16154       9359

The results are currently being presented as a simple histogram, where we just plot the number of users that took each action separately for each group (and I do see a difference in the distribution between groups).

Without presenting some uncertainty on these numbers though, I have no confidence that the results are actually due to a real difference in the experiment, or just some statistical uncertainty.

I know this is a super naive question but if I understand it correctly then to use the square root of the number of entries in each group as the statistical uncertainty then this would have to be a Poisson process, which implies that one of the events is rarer than the other. As you can see from the above, this isn't really the case and the split is around 60/40 or so between the two actions.

What I'd love to be able to do is say that X% of group Y took action A (or B), and be able to quote a +/- value for the uncertainty (or just with raw numbers rather than as a percentage).


  • 1
    $\begingroup$ Seems like a logistic regression problem. If so, you can confirm that Group 2 is 18% more likely (95% confidence interval: between 11% and 25% more likely) to do action A than B. $\endgroup$ – conjugateprior Dec 30 '16 at 23:00
  • 1
    $\begingroup$ Assuming you've got a data.frame with columns, a, b, and group then mod <- glm(cbind(a, b) ~ group, family=binomial, data=df) gets you coefficients interpretable in terms of logits of the probability of action a, and exponentiating them gets you odds, as per the wiki page. So exp(coef(mod)) gets you 1.18 = 18% increase, and exp(confint(mod)) gets you upper and lower bounds. $\endgroup$ – conjugateprior Dec 30 '16 at 23:45
  • 1
    $\begingroup$ I was going to say that, for this problem, choosing logistic regression over a chi-square test or a test of independent groups' proportions would be like getting out the snowblower when all you need to do is shovel 10 feet of walkway. But @anthr you seem to be handling the snowblower just fine :-) $\endgroup$ – rolando2 Dec 30 '16 at 23:52
  • 1
    $\begingroup$ @rolando2 Well, he does want intervals around things, and I'm always keen to give people more rather than less general tools. Strategically speaking, I also bet there are covariates lurking in the back of this problem. There almost always are :-) $\endgroup$ – conjugateprior Dec 30 '16 at 23:58
  • 1
    $\begingroup$ FYI, CIs in probability space: g <- data.frame(predict(mod, se.fit=TRUE)) to get them in logit space, then g$prob <- 1/(1+exp(-g$fit)) to get probabilities, g$lwr <- 1/(1+exp(-(g$fit - 1.96*g$se.fit))) to get lower, and g$upr <- 1/(1+exp(-(g$fit + 1.96*g$se.fit))) to get upper limits on them. R doesn't really make this easy... $\endgroup$ – conjugateprior Dec 31 '16 at 0:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.