3
$\begingroup$

Suppose I empirically estimate the standard error of some statistic to be 10% (perhaps I do this by bootstrapping).

Now, I want to know how much I need to increase the sample size to reduce the error to 5%. The estimate and sample are arrived at through complex procedures, and theoretically determining the relationship between sample size and standard error is not feasible. However, I believe that the standard error decreases as sample sizes increases.

Is it plausible to assume that standard error is proportional to the inverse of the square root of n (based on the standard error of a sample mean using simple random sampling)?

se = s / sqrt( n )

Do standard errors behave (very) roughly in this way in general in relation to sample size, regardless of the estimate and the sampling procedure? How bad of an assumption is this?

$\endgroup$
4
$\begingroup$

regardless of the estimate and the sampling procedure?

No, you will not have a "root-n" effect regardless of those things, since at least some standard errors do not scale with $\sqrt{n}$.

Many do -- quite possibly all the ones you will be likely to use -- but that's not all of them.

For things that do scale with $\sqrt{n}$ then you expect to halve the standard error by quadrupling sample size. So (at least if we're ignoring sampling variation in the estimate of $\sigma$), that's probably what you need.

One example of something that isn't proportional to $\frac{1}{\sqrt{n}}$ is the standard error of a kernel density estimate when the bandwidth is itself chosen as a function of $n$. [For some common choices of bandwidth formula the standard error goes down as $n^{-2/5}$ instead.]

$\endgroup$
  • $\begingroup$ Interesting. Even for your example where the error is proportional to n^(-2/5), n^(-1/2) is a good enough approximation for my purposes. Thank you. $\endgroup$ – Iggy25 Mar 8 '16 at 15:11
  • $\begingroup$ @Iggy25 Off the top of my head I can think of examples that scale as $n^{-1}$ and $n^{-1/3}$ that you could run into with real data analysis problems. It really depends on what classes of problems you can encounter. Even the suitability of ignoring the difference between $n^{-1/2}$ and $n^{-2/5}$ depends on what you're doing; I'd regard the situations in which I'd be prepared to ignore that difference as pretty limited. $\endgroup$ – Glen_b Mar 8 '16 at 22:12
3
$\begingroup$

Yes, generally, more samples would lead to smaller standard error. However, since adding more data is generally a random process, the decrease in standard error is not monotonic.

Consider the special case of Monte Carlo. Suppose you draw i.i.d samples from a distribution $F$, $X_1, X_2, \dots, X_n$. Suppose $Var_F(X_1) = \sigma^2$ and the mean is $E_F(X_1) = \mu$. Consider the task of estimating $\mu$ with the sample mean. Then, $$ \hat{\mu} = \dfrac{1}{n} \sum_{i=1}^{n} X_i.$$

The variance of $\hat{\mu}$ can be estimated having obtained the sample variance, $$\hat{\sigma}^2 = \dfrac{1}{n-1} \sum_{i=1}^{n}(X_i - \hat{\mu})^2. $$

Note that $\hat{\sigma}^2 \overset{a.s.}{\to} \sigma^2$ as $n\to \infty$. Due to this convergence, with more samples, we keep bettering our estimate of $\sigma^2$, and the general trend is a decrease in the standard error $\hat{\sigma}/\sqrt{n}$.

However, if we add a new observation $X_{n+1}$ that lies in the tails of $F$, it is possible for that the new estimate of the variance $\sigma^2$ is larger than the old estimate, so that $\hat{\sigma}/\sqrt{n+1}$ is larger that the previous standard error.

If I understand the rest of your question correctly, you want the standard error to reduce from 10 units to 5 units. Suppose you have $n_1$ samples that gave you a standard error of 10 units. At the cost of being very approximate, lets equate 10 to the standard deviation of the estimate.

$$10 = \dfrac{\sigma}{\sqrt{n_1}} $$

You want to find $n_2$ so that $$5 = \dfrac{\sigma}{\sqrt{n_2}} $$

Dividing the two equations,

\begin{align*} 2 & = \sqrt{\dfrac{n_2}{n_1}}\\ \end{align*}

and now you can solve for $n_2$ since you know $n_1$. Of course, this will be an approximate sample size calculation since we used the truth, $\sigma$.

$\endgroup$
  • $\begingroup$ This is right, but it doesn't quite answer my question. Yes, it is possible for the estimated standard error to increase as sample size increases as per your explanation. However, the true standard deviation of the estimate (I think) cannot decrease. What I really want to know is, if my current standard error is 10% how many observations do I need to add before I can expect it to drop to 5%, and is my approach a reasonable way of getting this number. $\endgroup$ – Iggy25 Mar 7 '16 at 19:24
  • $\begingroup$ Can you explain the usage of %s when using standard error. In other words, you say standard error of 10%, but 10% of what? $\endgroup$ – Greenparker Mar 7 '16 at 19:31
  • $\begingroup$ Sorry, I can see how that is confusing. My statistic itself is a ratio, and is expressed as a percentage. So I have been thinking of the standard error as a percentage as well. I am asking about reducing the standard error by some ratio - ie from X to 0.5X. $\endgroup$ – Iggy25 Mar 7 '16 at 22:23
  • $\begingroup$ I have edited the answer, and hopefully this answers your question. $\endgroup$ – Greenparker Mar 8 '16 at 7:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.