Asymptotic T-test validity for proportion values

Question

I have a large sample of values, bounded in $[0, 1]$, divided into two conditions A and B, and I want to test the significance of one condition A having higher value than condition B.

For the details, these values are bootstrap scores for a given relationship (A or B) of species taxa in many different phylogenetic gene trees (i.e. A = taxon1 is sister to taxon2 VS B = taxon1 is sister to taxon3). I would like to set aside the fact that actual phylogenetic tests exist for this kind of question and come back to the case of a t-test for proportion values.

I feel that a t-test (Welch) for independent samples would not be valid, even if the sample is very large, because values are proportions (note: I don't have the original counts).

Indeed, on this forum many people suggested (here, here or here) not using T-test but something more appropriate like a generalized linear model, which was my intuition as well.

However, I could not answer: which assumption of either the Central Limit Theorem or of the Welch's T-test is violated by my dataset?

First, I reasoned by translating the T-test problem into the (I believe equivalent) linear regression problem, in which we would clearly violate the assumption of identically normally distributed residuals. My problem is that I still don't find what is violated in the T-test formulation. How are the assumptions related between these two formulations?

Indeed, it seems that with the Central Limit Theorem, the mean of many percentages would follow a normal distribution, thus the condition that the mean is normally distributed would hold; so the violated assumption has to be another one in the T-test. I saw that the $T$ variable

$$ T = \frac{Z}{\sqrt{U/k}} $$

requires that $Z$ and $U$ be independent to follow the Student distribution. Is it the assumption that is violated by a dataset of proportion values?

Answer 1

The answer is that, contrary to popular belief, the Welch (not Welsh) T-test kind of assumes equal variance of populations. Now, before you go crazy and think you were lied to, let me explain. A T-test only holds when the following are true:

1. When the two populations have equal variances (in practice: similar variances).
2. The distributions are perfectly normally distributed (In practice: The skewness must be small, or the sample size must be very large to make up for it).

Welch's T-test applies a correction to the regular t-test that makes violations of the first assumption practically unimportant when condition 1 is fulfilled. However, the Welch t-test, in theory, is still not exact unless condition 1 is fulfilled or the sample size is very large.

In theory, the Welch and regular T-tests have their assumption of equal variances violated when dealing with a binomial distribution, because a binomial distribution has a standard error equal to: $$\frac{\sqrt{p (1 - p)}}{\sqrt n}$$

Notice that this variance depends on $p$. Therefore, any change in $p$ technically invalidates any t-test, so a t-test can't be used.

But what if $n$ is really big? Well then, the variance hardly changes, because the variance is close to 0, and going from "Basically 0" to "Basically 0" is a change of, well, basically 0. So the fact that the variance changes a little is fine, because Welch's t-test is insensitive to small differences in variance (unlike the regular t-test).

Answer 2

The proposed answer here assumes that, despite your data being bounded on [0,1], the means (average proportions) for each condition are not too close to either zero or one.

In brief, if your sample size is sufficiently large, then use of the Welsh t-test is acceptable. (And if the sample size is truly large enough, you probably won't notice a difference in the P-values between a 2 independent sample t-test and the Welsh t-test.)

So, the major concern is that your data (the proportions for varying sample sizes) is being drawn from potentially very non-normal distributions.

Here is a small bit of code that helps demonstrate this:

grp1 <- rep.int(NA,5000)
grp2 <- rep.int(NA,5000)

for(ctr in 1:5000) {
  grp1[ctr] <- sum(sample(c(0,1),50,replace=T))/50
  grp2[ctr] <- sum(sample(c(0,1),2000,replace=T))/2000
}

hist(c(grp1,grp2),breaks=100)

In brief, what you see in this plot is two normal distributions on top of each other. As there is such a distribution for each different sample size, you can see why the "combination" of all of these possible distributions would result in a non-normal distribution. (Though, all the distributions would have the same mean, if not the same variance, and thus you would still have a relatively symmetric distribution.)

However, with large enough samples drawn from these non-normal distributions, the mean of the samples will follow a normal distribution (by the CLT). To clarify, the CLT requires the same sample size for the sampling distribution, but it is the population distribution that bears the burden of having different samples for each proportion calculated in this context. The sampling distribution would be based on the number of samples in conditions A and B (not the sample sizes used for the data collection process in each observation in each condition).

Thus, assuming you do indeed have sufficiently large enough samples, the Welsh t-test should be robust enough to provide a reasonable estimate for the P-value under a null hypothesis that the population means of the two conditions are the same.