1
$\begingroup$

So, the idea of a two sample t-test is pretty simple. Two samples, each with values drawn from a given set. Consider the difference in the means of the two sample - the mean and variance of its distribution. Then, calculate the t-statistic and finally the p-value on the assumption that the difference follows a t-distribution. When the two samples consist of integers for example, this seems reasonable because the difference can be any integer between -Inf and +Inf (and the mean can be rational). But what about cases where the two samples consist of just binary outcomes? The link here - http://stattrek.com/hypothesis-test/proportion.aspx suggests doing a permutation test. The only difference is in the way they calculate the variance. Apart from that, they still use a normal distribution on the z-score for the p-value. However, the difference between the means in the two samples can never be greater than 1. A normal distribution however, lives in [-Inf,+Inf] is it therefore appropriate to use it to calculate the p-value? Similarly for cases where the two samples consist of only fractional values (all between [0,1]). How should the p-value be calculated without an assumption of normality since the distribution isn't normal (max value 1).

$\endgroup$
1
  • $\begingroup$ Can you show me where on that page you link to it suggests a permutation test? I tried searching but couldn't locate 'permutation test', 'randomization test' or 'resampling test' on the page. $\endgroup$ – Glen_b Apr 25 '14 at 12:14
1
$\begingroup$

Means are scaled sums.

Sums of independent Bernoulli(p) random variables are binomial(n,p).

A standardized binomial(n,p) can be approximated by a standard normal if n is large and p is not close to 0 or 1. Consequently normal approximations to unscaled binomials (or binomials scaled to proportions) work as well:

enter image description here

The fact that a binomial proportion is restricted to [0,1] while a normal is not is not of much consequence, since if n is sufficiently large, the amount of probability that the approximating normal has outside the range on the binomial is so small as to be of little consequence to the approximation.

If the normal approximation is not sufficiently accurate, you can just work directly with the binomial distribution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.