I have run an experiment in which I'm comparing the behavior of two groups of unequal size (
group2), where each subject in the group has to take one two possible actions. So I'm ending up with data of the form:
action_A action_B group1 3320 2273 group2 16154 9359
The results are currently being presented as a simple histogram, where we just plot the number of users that took each action separately for each group (and I do see a difference in the distribution between groups).
Without presenting some uncertainty on these numbers though, I have no confidence that the results are actually due to a real difference in the experiment, or just some statistical uncertainty.
I know this is a super naive question but if I understand it correctly then to use the square root of the number of entries in each group as the statistical uncertainty then this would have to be a Poisson process, which implies that one of the events is rarer than the other. As you can see from the above, this isn't really the case and the split is around 60/40 or so between the two actions.
What I'd love to be able to do is say that X% of group Y took action A (or B), and be able to quote a +/- value for the uncertainty (or just with raw numbers rather than as a percentage).