I have a question about something that is probably very basic to statistics but I feel I don't fully understand

I've found that generally it is much harder to get high power (smaller) confidence intervals with binomial data (0,1s)

When I want to calculate the difference between two proportions it is hard to get a high p value and confidence intervals. However, with studies with the same amount of participants for which I can calculate means (because the variables are measured continuously rather than as 0 or 1 choice) the power is much higher. I think I have a good sense of why this is but i'd like someone to explain it to me in a basic way so that I can check my understanding.

Is there are graph which anyone can point me to (or some general rule) that links sample size with power when using binomial vs continuous data. I feel like I should avoid collecting binary outcomes where ever possible because of the lack of power. Is this correct

Additional comment added 11.05.2014 12:04

TO CLARIFY: My question is - if we are measuring exactly the same thing but with 2 different ways of collecting data. One is binary (e.g. do you prefer B 'more' or 'less' than A), one is continuous (do you prefer A or B. Please rate preference for A on scale of 0-1, where B is 0.5), then do I get higher power with the continuous measurement method - and if so why? I seem to, because I get a mean scores from the continuous method rather than proportions, and the std error of the mean scores generally seem to be lower than that of the proportions, when expected values are the same (e.g mean of 0.5, proportion of 0.5), and number of participants doing the test is the same.

  • $\begingroup$ How do the effect sizes compare? (Indeed, how does one compare effect sizes across the two?) What is $p$ in the binomials, typically? What is $\sigma^2$ in the measured variables, typically? Are they both paired or both unpaired, or one of each? $\endgroup$ – Glen_b -Reinstate Monica May 7 '14 at 22:34
  • $\begingroup$ this is hypothetical data but i'd looking to find the same size effect with both binary measurement method and continuous measurement. The p in the proportions data can be anywhere from 0-1. I'm not sure what the variance would be but as both binary measures and continuous measures are both measuring the same underlying thing - i'd expect that the underlying variance is the same and it's just a matter of what measuring as binary or continuous does to the measured variance. $\endgroup$ – user45114 May 8 '14 at 8:42
  • $\begingroup$ If you're measuring different things (like proportions vs some measurement) why on earth would they have the same variance? $\endgroup$ – Glen_b -Reinstate Monica May 8 '14 at 9:23
  • $\begingroup$ Hi, its the same construct being measured in both cases.... but i'm either using a categorical measurement or a continuous measurement (i.e how much more pleasant is thing a than X - 'less pleasant or 'more pleasant' vs if x is 50 rate A on a scale of 0 - 100. $\endgroup$ – user45114 May 11 '14 at 9:24
  • $\begingroup$ Why would the values of those random variables have the same variance? $\endgroup$ – Glen_b -Reinstate Monica May 11 '14 at 9:29

A Bernoulli (binary) random variable has one bit of information. This is the lowest amount of information something can have short of no information at all. A continuous response that is integer valued 0-100 has 6 bits (binary digits) of information. Assuming that not all of the 6 bits are pure noise, the non-binary variable has very much more information than the binary one. This results in better statistical power and precision when making inferences.

  • $\begingroup$ Thanks. That helps. I'm still having trouble seeing how this relates to higher p values (or smaller confidence intervals) for a fixed number of people in the study. Is it something to do with the fact that with more values (a continuous measure) we can estimate the std error, based on our sample data - but with only 0,1s we cannot $\endgroup$ – user45114 May 18 '14 at 19:08
  • $\begingroup$ No. You didn't fully read the last answer. More statistical information means more precision and power. More unique values of $Y$ that are not pure noise means more information. $\endgroup$ – Frank Harrell May 18 '14 at 19:11
  • $\begingroup$ Sorry for not being clear. Am i getting any closer to understanding with this line of thinking: I understand that i get more information from the continuous measure. I suppose I get more statistical power with this high resolution because I can work out a precise point on a normal distribution where each score lies (for any single person's score), which gives me a measure of distance of each score from the null hypothesis. I immediately get statistical data on how likely the score is... $\endgroup$ – user45114 May 18 '14 at 19:55
  • $\begingroup$ ... However, With 0/1 I can't place each persons score along the normal distribution, but over a number of scores a can calculate a probably (or percentage) which i can place on a normal distribution to get a statistical measure of how likely that overall percentage is. - but I need more people to get to this stage. $\endgroup$ – user45114 May 18 '14 at 19:56
  • $\begingroup$ This has very little to do with the normal distribution. Think of this as an example. You are told that two runners each ran a mile in under 6 minutes. You want to pick the best runner for your team. Your decision will not be optimal without knowing exactly how fast they ran. Better information leads to better decisions. $\endgroup$ – Frank Harrell May 18 '14 at 20:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.