Can I compute a confidence level for a z-test based on the size of the samples? I'm doing a two sample z-test for the difference between means using pooled variance.
Suppose I've run two experiments.  In the first experiment, the sample size for the two populations is only 100, and I get a z-value of 5.0.  In the second experiment, the sample size for the two populations is one million, and I also get a z-value of 5.0.  Can I quantify somehow a different confidence level or confidence interval for z-values from the two experiments?
I'm hoping there's some generally applicable quantitative formula, not just general guidelines about "sample sizes being sufficiently large".
 A: Yes, this is essentially what the t-test does.  While a z-score represents a standardized difference between two scores (or a score and a mean) based on the population standard deviation (X1 - X2 / SD(1,2)), a t-value represents the difference between two means adjusted for their pooled standard error.  Standard error reflects sample size.  Here's the "t", where X's are means and SE is standard error of the distribution of the means:
X1 - X2 / SE(1,2)
The Student t distribution is a distribution of sampling errors of the mean given X degrees of freedom.  Above approximately 30 subjects in a sample, the Student t distribution converges with the normal distribution (i.e., z-scores) such that a z-score of 2.0 ~= a t-value of 2.0 and both are significant to p < .05.
A: I don't know if you can quantify the confidence level of the z-values, but you can certainly give a smaller confidence value for your estimates of the mean.
Read the wiki page on standard error. Standard error differs from standard deviation in that standard error is the amount of variation in the parameter estimates. In this case, the standard error should be calculated as
$\hat \sigma_\mu = \frac{\hat \sigma_x}{\sqrt{N}}$
where $N$ is the sample size. As such, your estimates for the mean will have much tighter bounds when you calculate a confidence interval (e.g., $95\% CI = \hat{\mu} \pm 1.96\hat{\sigma}_\mu$). As $N$ increases, the bounds will get tighter.
Now, if you're dead set on finding an estimate for the z-score, we can find error bounds for it. Assuming you know the population mean $\mu$ and variance $\sigma$ of the null distribution, we can define a new random variable, $Z$:
$Z = \frac{X- \mu}{\sigma}$
We can find the standard error or error depending on which you'd prefer. It sounds like you're more interested in finding the standard error, so let's look at
$ \hat{\mu}_Z = \frac{ \hat{\mu}_X- \mu}{\sigma} $
$Var(\hat{\mu}_Z) 
= Var(\frac{\hat{\mu}_X- \mu}{\sigma})
= \frac{Var(\hat{\mu}_X) - Var(\mu)}{\sigma^2}
= \frac{Var(\hat{\mu}_X)}{\sigma^2}$
Since $Var(\hat{\mu}_X) = \frac{\hat{\sigma}_X}{\sqrt{N}}$, the standard error for a mean z-score is
$Var(\hat{\mu}_Z) = \frac{\hat{\sigma}_X}{\sqrt{N}\sigma}$
You can generalize to two samples as well; it all comes down to basic variance calculations. Bear in mind this is from calculating a z-score from a sample mean, you can also derive things from working with Z. Bottom line is divide by sigma for most of these z normalizations.
