I have two experiments E1 and E2. Each consists of several trials (number of trials is different between experiments). Each trial has the binary outcome: 0 or 1. So, for example, i have (real sizes are hundreds):

E1 = [0 1 1 1 0 0 1 0];
E2 = [1 1 0 0 1];

I want to compare the probability of having 1 in both experiments. I have an idea, just to calculate the mean and SEM (sqrt((p)(1-p)./n)) and plot it as a bar plot.

My question is, whether I am correct with mentioned above way of data visualization, and which test I should use to find the significance of differences?

up vote 2 down vote accepted

Let's say you've used R to organize your data:

theme_set(theme_bw()); theme_update(plot.title = element_text(hjust = 0.5))

# artificially increase sample size by a factor of 10 for realism.
e1 <- rep(c(0,1,1,1,0,0,1,0), 10)
e2 <- rep(c(1,1,0,0,1), 10)

# prepare a summary table with one row per experiment
df <- tibble(
  experiment=factor(c( rep(1, length(e1)), rep(2, length(e2)))), 
  outcome=c(e1, e2)) %>%
  group_by(experiment) %>% 
  summarize(n=n(), p=mean(as.numeric(as.character(outcome)))) %>%

The df data.frame contains a row for each experiment, with sample size n, estimated parameter p, and se for standard error.

# A tibble: 2 x 4
  experiment     n     p     se
  <fct>      <int> <dbl>  <dbl>
1 1             80   0.5 0.0559
2 2             50   0.6 0.0693

Then a good, easily understood, and visually faithful way to present the data is as two side-by-side bars with error bars overlayed:

# visualizing with barplot + errorbar
ggplot(data=df, aes(x=experiment, y=p, fill=experiment)) + 
  coord_cartesian(ylim=c(0,1)) + 
  geom_bar(stat='identity') +
      ymax = p + 1.96*se)
  ) +
  ggtitle("Comparison of Experiments")

barplot with error bars

Visually, this is easy to interpret because the error bars show the 95% confidence interval for the "true" height of bar; therefore, if the confidence intervals for two bars overlap, the difference is probably not significant. (But we can be more formal about that too, see below.) In my opinion, it's extremely important to show confidence intervals for proportions and seriously misleading to show a bar chart without confidence intervals because the CI is much wider for small n than most laymen realize. It can take tens of thousands of independent trials before a probability can be estimated to within 1%. For smaller sample sizes, a visually "obvious" difference between two bars is in fact likely to be statistically insignificant; the cure is to make the width of the confidence intervals visually apparent, as we have done.

An alternative visualization to the bar plot is to use a "box-and-whisker" plot, or just boxplot for short. The "standard" boxplot is defined in terms of medians and quantiles so isn't very useful for binary data. But we can use the same visualization to display confidence intervals derived from our normal approximation.

# visualizing with (slightly non-standard) boxplot
ggplot(data=df, aes(x=experiment, y=p, fill=experiment)) + 
  coord_cartesian(ylim=c(0,1)) + 
      lower = p - se,
      middle = p,
      upper = p + se,
      ymax = p + 1.96*se)
  ) +
  ggtitle("Comparison of Experiments")

boxplot with whiskers

I prefer this visualization because it's easier (for me at least) to see if the confidence intervals overlap at all.

The confidence interval used in the above visualizations is the "normal approximation" version, justified because your example p are close to 0.5 and your sample size is in the hundreds. But you should be aware there are many other ways to calculate this confidence interval, but this chiefly becomes important when sample size is small or p is very close to 0 or 1 in which case the CI is not even symmetric!

Finally, you asked about which statistical test to use; Since you have sample sizes in the "hundreds", the best is also the simplest: the z-test for equal proportion for two groups. It uses a test statistic with a normal distribution:

$$Z=\dfrac{(\hat{p}_1-\hat{p}_2)-0}{\sqrt{\hat{p}(1-\hat{p})\left(\dfrac{1}{n_1}+\dfrac{1}{n_2}\right)}} , \hat{p}=\dfrac{Y_1+Y_2}{n_1+n_2}$$

Note that this test is "pooling" the variance from both experiments together into a single estimate of standard error, so the denominator is a little different than the S.E. formula you used above which is only valid for one experiment at a time. This test can break down if p is very close to 0 or 1, so check the test assumptions carefully.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.