How can sample SD be substituted for population SD in standard error of means formula?

Question

Standard error of sampling distribution of means is generally defined as (population SD/sqrt(n)) where n=sample size. But where population parameters are not known I have seen sample SD being used in the numerator in place of population SD. But I believe this will result in a SE that is drastically different.Since obviously a sample SD will be much different from population SD. Can you explain this intuitively how this works? It would make more sense for me if, in the absence of population parameters which are unknown, we draw a large sample (with n>>>30), and then use the SD of that sample inthe numerator. But sample size will have to be really high(how high I do not know quantitatively). But even with this approach your SE is only an approximate value, not an accurate one

Answer 1

If you have a random sample of size $n$ from a normal population with known standard deviation $\sigma,$ then the standard error of the sample mean, $\bar X =\frac 1n\sum_{i=1}^n X_i,$ is $\sigma/\sqrt{n}.$

Then a 95% z confidence interval for $\mu$ is of the form $\bar X \pm 1.96\frac{\sigma}{\sqrt{n}},$ where $\pm 1.96$ cut probability from the upper and lower tails, respectively, of a standard normal distribution.

If $\sigma$ is unknown and estimated by the sample standard deviation $S=\sqrt{\frac{1}{n-1}\sum_{i=1}^n (X_i - \bar X)^2},$ then the (estimated) standard error of $\bar X$ is $S/\sqrt{n}.$ [Because, upon seeing $S$ in place of $\sigma,$ one knows we're dealing with an estimate, the word estimated is often omitted.]

Then a 95% t confidence interval for $\mu$ is of the form $\bar X \pm t^*\frac{\sigma}{\sqrt{n}},$ where $\pm t^*$ cut probability from the upper and lower tails, respectively, of Student's t distribution with degrees of freedom $\nu = n-1.$

In practice $t^* > 1.96,$ making the t confidence interval somewhat longer than the "corresponding" z CI. This extra length takes into account that the sample standard deviation $S$ is only approximately equal to $\sigma.$

For a small sample size, such as $n = 10,$ we have degrees of freedom $\nu = n-1 = 9$ and $t^* = 2.262 > 1.96.$ For samples of moderate size, such as $n = 30,$ we have $\nu = 29$ and $t^* = 2.04,$ close to $1.96.$ For large sample, such as $n = 200,$ we have $t^* = 1.972 \approx 1.96.$ [You can get these values of $t^*$ from a printed table or by using software, such as R as below.]

qt(.975, 9)
[1] 2.262157
qt(.975, 29)
[1] 2.04523
qt(.975, 199)
[1] 1.971957

As $n$ increases the sample standard deviation $S$ gets closer to the population standard deviation $\sigma,$ the estimated standard error of $\bar X$ gets more accurate, and the length of the 95% t CI gets closer to the length of a 95% z CI.

Suppose $\bar X$ is based on a normal population with $\sigma= 5$ of size $n=10.$ Then the margin of error of a 95% z confidence interval for unknown $\mu$ is $M = 1.96(5)/\sqrt{10} = 3.099$ and the width of the CI is $2M= 6.198.$ If $\sigma$ is unknown and we use the (highly variable) sample standard deviation $S$ to make a t confidence interval, then the margin of error will vary and will tend to be longer than $3.1$--on average $3.48,$ as shown in the R simulation below.

set.seed(224)
n = 10;  t.10 = qt(.975, 9)
E = replicate(10^5, t.10*sd(rnorm(n,50,5))/sqrt(n))
summary(E)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.6693  2.8994  3.4508  3.4812  4.0224  7.5266 

For $n = 30,$ the expected value of the margin of error for a 95% t CI is $E(M) = 1.852.$ And for $n = 200,$ we have $E(M) = 0.696.$ [The exact values of $M$ in a 95% z CI for $n = 30$ and $200$ are $M=1.789$ and $M=0.693,$ respectively]

set.seed(225)
n = 30;  t.30 = qt(.975, n-1)
E = replicate(10^5, t.30*sd(rnorm(n,50,5))/sqrt(n))
summary(E)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.8887  1.6838  1.8464  1.8520  2.0140  2.9576 

set.seed(225)
n = 200;  t.200 = qt(.975, n-1)
E = replicate(10^5, t.200*sd(rnorm(n,50,5))/sqrt(n))
summary(E)
    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  0.5443  0.6726  0.6963  0.6964  0.7198  0.8516 

For each sample size, the 95% z CIs and the 95% t CIs will cover (include) the population mean $\mu$ for 95% of samples, over the long run. For each sample size, the 'probability factors' $t^*$ are chosen to be sufficiently larger than $1.96$ to compensate for the variability of $S$ as an estimate of $\sigma,$ when $\sigma$ is not known.

The R program below illustrates that 95% of 100,000 z and t confidence intervals based on $n = 25$ normal observations cover the population mean $\mu.$ On average, the t CIs are a little longer.

set.seed(2021)
n = 25;  mu = 50;  sg = 5;  t.25 = qt(.975,24)
m = 10^5; LL.t = UL.t = LL.z = UL.z = numeric(m)
for(i in 1:m) {
 x = rnorm(n,mu,sg); a=mean(x); s=sd(x)
 LL.z[i] = a - 1.96*sg/5
 UL.z[i] = a + 1.96*sg/5
 LL.t[i] = a - t.64*s/5
 UL.t[i] = a + t.64*s/5
}

mean(mu > LL.z & mu < UL.z)
[1] 0.95046       # aprx 95%
mean(mu > LL.t & mu < UL.t)
[1] 0.95023       # aprx 95%

2 * 1.96*sg/5
[1] 3.92          # exact length of z CIs
mean(UL.t - LL.t)
[1] 4.086751      # average length of t CIs