# Can I compute a confidence level for a z-test based on the size of the samples?

I'm doing a two sample z-test for the difference between means using pooled variance.

Suppose I've run two experiments. In the first experiment, the sample size for the two populations is only 100, and I get a z-value of 5.0. In the second experiment, the sample size for the two populations is one million, and I also get a z-value of 5.0. Can I quantify somehow a different confidence level or confidence interval for z-values from the two experiments?

I'm hoping there's some generally applicable quantitative formula, not just general guidelines about "sample sizes being sufficiently large".

-
 Correction: I believe the problem is not sample size, as originally posed, but rather unequal sample sizes and unequal variances. I believe what happens is when I perform a large number of tests, some feature may correlate by chance with the feature of interest. In this case, the size of the matching samples is small, and the non-matching is large. Also, the variance in the matching samples is small. Apparently, my z-test fails in this scenario. What is the best method to screen for these random occurrences? – Tyro Jul 15 '12 at 21:51

Yes, this is essentially what the t-test does. While a z-score represents a standardized difference between two scores (or a score and a mean) based on the population standard deviation (X1 - X2 / SD(1,2)), a t-value represents the difference between two means adjusted for their pooled standard error. Standard error reflects sample size. Here's the "t", where X's are means and SE is standard error of the distribution of the means:

X1 - X2 / SE(1,2)

The Student t distribution is a distribution of sampling errors of the mean given X degrees of freedom. Above approximately 30 subjects in a sample, the Student t distribution converges with the normal distribution (i.e., z-scores) such that a z-score of 2.0 ~= a t-value of 2.0 and both are significant to p < .05.

-
 I don't understand how the t-test helps me quantify the confidence level any more than the z-test does. If I run two experiments with vastly different sample sizes and both experiments yield the same t-value, are you saying that the confidence level in both tests is the same? – Tyro Jul 13 '12 at 22:09 Can someone explain why this answer was up-voted? – Tyro Jul 14 '12 at 2:12 Given the t-value you calculate AND the number of subjects/datapoints in your sample, you can find out in the Student t distribution what the probability is of finding a difference this large by chance given that both samples are drawn from the same population. THIS is the "confidence level" you need (otherwise known as a probability value, p-value, or alpha). – user268208 Jul 14 '12 at 7:26 Yes, the p-value gives you a confidence on how likely a z score of 5 is under the null hypothesis (that the populations are the same). Given a z-score of 5, it is very likely they are differently distributed (something like 99.9999% likely, I'm on a mobile device and don't have my stats program). – Oliver Jul 14 '12 at 21:55 Right, I know that's the standard methodology. It's just that I've performed a very large number of random simulations with known associations (see my comment in the other reply below), and there seems to be a strong correlation between false-positives and small sample sizes. That's why I'm asking if there is a quantitative relationship between sample size and confidence level for a z-value. – Tyro Jul 15 '12 at 0:56
show 1 more comment

I don't know if you can quantify the confidence level of the z-values, but you can certainly give a smaller confidence value for your estimates of the mean.

Read the wiki page on standard error. Standard error differs from standard deviation in that standard error is the amount of variation in the parameter estimates. In this case, the standard error should be calculated as

$\hat \sigma_\mu = \frac{\hat \sigma_x}{\sqrt{N}}$

where $N$ is the sample size. As such, your estimates for the mean will have much tighter bounds when you calculate a confidence interval (e.g., $95\% CI = \hat{\mu} \pm 1.96\hat{\sigma}_\mu$). As $N$ increases, the bounds will get tighter.

Now, if you're dead set on finding an estimate for the z-score, we can find error bounds for it. Assuming you know the population mean $\mu$ and variance $\sigma$ of the null distribution, we can define a new random variable, $Z$:

$Z = \frac{X- \mu}{\sigma}$

We can find the standard error or error depending on which you'd prefer. It sounds like you're more interested in finding the standard error, so let's look at

$\hat{\mu}_Z = \frac{ \hat{\mu}_X- \mu}{\sigma}$

$Var(\hat{\mu}_Z) = Var(\frac{\hat{\mu}_X- \mu}{\sigma}) = \frac{Var(\hat{\mu}_X) - Var(\mu)}{\sigma^2} = \frac{Var(\hat{\mu}_X)}{\sigma^2}$

Since $Var(\hat{\mu}_X) = \frac{\hat{\sigma}_X}{\sqrt{N}}$, the standard error for a mean z-score is

$Var(\hat{\mu}_Z) = \frac{\hat{\sigma}_X}{\sqrt{N}\sigma}$

You can generalize to two samples as well; it all comes down to basic variance calculations. Bear in mind this is from calculating a z-score from a sample mean, you can also derive things from working with Z. Bottom line is divide by sigma for most of these z normalizations.

-
 Yes, and since the z-value depends on the estimated means, it seems to me that there should be some formula that "propagates" this confidence interval for the means into the z-value. Also, I would think that there should be a confidence interval for the standard deviation that would also "propagate" to the z-value. – Tyro Jul 13 '12 at 22:07 Are you sampling from both populations? Do you know the value of either population mean or either population variance? I will add to my answer as I have a solution, but it depends on the situation. – Oliver Jul 14 '12 at 2:02 Yes, I am sampling from both populations. I calculate the mean and the variance from the samples; I do not know the mean or variance for the population as a whole. Basically, I find the 2x2 contingency table from the samples, and calculate the z-value from the table. – Tyro Jul 14 '12 at 2:10 I've added a solution for if you know the mean and variance for a second population and are normalizing by using that. It should give you a skeleton of how to solve your problem with two samples. I can go through the math for two samples if my answer is confusing or hard to generalize. – Oliver Jul 14 '12 at 2:21 Although, I'm still a little confused why you need error bounds or confidence on your z-scores. Usually, working with means is more interpretable. Can you elaborate on what your aim is and how you're calculating your z-scores (e.g., are you calculating them from the raw observations or from a mean)? – Oliver Jul 14 '12 at 2:36