2
$\begingroup$

I have a question as to whether the one-sample t test or the one-sample z test is appropriate in a sample of n=17 BUT with a known population variance.

I would usually use a one-sample t-test for such small samples. However, since I have a reference variance from another independent sample I was wondering if maybe the one-sample z-test would be appropriate.

I have conducted both and have obtained some significant effects. I thought that the tests would be underpowered given the small sample size. How should I interpret the results now? Are the results still taken to be seriously?

Thank you everyone!

$\endgroup$
7
  • 2
    $\begingroup$ It sounds like you're using the variance from another sample when calculating your test statistic. That's not "a known population variance" it's a sample variance (albeit from a different sample). Can you clarify your question to be consistent with whatever the actual situation is? (to either be talking about a known population variance or about a sample variance from another sample) $\endgroup$
    – Glen_b
    Dec 5 '19 at 12:56
  • $\begingroup$ Our statistics professor has always been telling us that if we have a variance from another independent sample we can treat it as a known "population" variance. That's why I was talking about the population variance. $\endgroup$
    – Milkovena
    Dec 5 '19 at 17:03
  • $\begingroup$ Do you get different results with t-test and z-test? 17 is not a such a small sample. $\endgroup$ Dec 5 '19 at 18:07
  • 1
    $\begingroup$ @Milkovena Interesting. Outside of the circumstances where it would be reasonable to treat an insample standard deviation as the population s.d. (i.e. very large sample size), your professor is wrong. I'll come back and write up an answer explaining how to correctly use a sample variance from a different sample (given suitable assumptions) $\endgroup$
    – Glen_b
    Dec 6 '19 at 1:48
  • $\begingroup$ Please add the additional context / information you gave in comments (about your professor's claim) to the body of the question, since it's important to an answer. $\endgroup$
    – Glen_b
    Dec 6 '19 at 4:26
1
$\begingroup$

Imagine we have the following situation:

A previous sample of size $m$, from which we have an estimate $s^2_m$ of its population variance $\sigma^2$. (Here the $m$ subscript is simply denoting the size of the first sample, rather than the denominator in the variance calculation. It's just there to remind us that it's not calculated on the current sample.)

A current sample of size $n$ from which we have an estimate $\bar{y}$ of its population mean $\mu$. We assume here that the variance of the population from which the current sample was drawn is also $\sigma^2$.

We assume both populations are normally distributed. We assume independence within and between groups.

For a one sample test of $H_0$: $\mu = \mu_0$ (against one or two sided alternatives in the usual fashion), we can form the statistic $t=\frac{\bar{y}-\mu_0}{s_m/\sqrt{n}}$. This will have a t-distribution with $m-1$ d.f. (NB $n$ doesn't come into the df at all)

For a two sample test of equality of the population means of the two samples described above, we can form a test statistic whose numerator is the difference in their sample means and whose denominator is $s_m \sqrt{\frac{1}{m}+\frac{1}{n}}$, and again the d.f. is $m-1$.

This result is relatively straightforward mathematically, but we can check it via simulation. Here's an example done in R. Assume $m=10$ and $n=5$ (I choose small samples so that we can easily distinguish the distributions). Let's demonstrate that the null distribution of the statistic for the first case above is consistent with the claim that the distribution is $t_9$ (i.e. $t$ with $m-1$ df). We simulate the t-test statistic and compare the distribution of under the two claims (your professor's and mine). However, it can be hard to visually distinguish a t and a normal distribution. Consequently, I will convert the distribution of the t-statistic into the corresponding p-values under the assumption that the statistic has a normal distribution and the assumption it has a $t$ distribution with 9 df. If one of us is correct, the p-values from our claimed distribution should look uniformly distributed between 0 and 1. I will also look at the achieved type I error rate (i.e. actual significance level) for a 5% two tailed test under both approaches.

Here's the R code to do that for the one-sample test:

nsim = 100000
sigma = 1 # without loss of generality but feel free to change things
m = 10
n = 5
mux = 20 # ditto
muy = 10 # ditto, demonstrating these don't have to be the same
tsim = replicate(nsim,{
  x = rnorm(m,mux,sigma)
  sm = sd(x)
  y = rnorm(n,muy,sigma)
  ybar = mean(y)
  (ybar-muy)/(sm/sqrt(n))
})
opar=par()
par(mfrow=c(1,2))
hist(pt(tsim,9),n=100)   # transforming test stat. to p-values if it's t9
hist(pnorm(tsim),n=100)  # ditto, if it's normal
par(opar)

Which gives:

histogram of a probability integral transformed test statistic under both my claim and your professor's claim

As you see, my p-values (on the left) are plausibly close to a uniform while your professor's p-values (on the right) are definitely not. This doesn't prove it's t9, but it does demonstrate that it's not normal, and that it can't be very far from a t9. My claim remains plausible.

Let's seem how much this would affect the significance level of a two-tailed t-test that we wish to be at the 5% level:

mean(pt(tsim,9)<.025|pt(tsim,9)>.9755)  # my claim
[1] 0.05011  # about 5%; within sampling variation 

mean(pnorm(tsim)<.025|pnorm(tsim)>.9755)  # your professor's claim
[1] 0.08228  # 8.2% - this is quite far out

So you should do the t-test but use as the df that for the sample standard deviation you used in the denominator (i.e. one less than the sample size in the original sample your variance estimate came from).

We could do a simulation for the two sample case in similar fashion; again it's $m-1$ df. You'd need to make the population means the same, calculate both sample means and then set the statistic to
(xbar-ybar)/(sm*sqrt(1/m+1/n)).

So the problem with doing a z-test instead of a t-test in this situation is that your significance level is inflated (just as it is if you did a z-test instead of a t-test with the s.d. calculated from the current sample). This lifts both power and the type I error rate (potentially a lot, as we saw in the simulation).

Naturally, if $m$ is quite large, you can get a reasonable approximation from treating the t-statistic as normal, but there's no reason to avoid doing the t-test if the assumptions are reasonable (why approximate what is easily done exactly?)

$\endgroup$
2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.