0
$\begingroup$

I have a very large dataset (with > 2 million simulated values). I want to compute standard error for this dataset. To do that, I divide the standard deviation by square root of number of observations. However, because of the large number of observations, the standard error is quite low. Is there a way to subsample instead and compute standard error?

$\endgroup$
  • 1
    $\begingroup$ At first glance, it's not obvious why the standard error calculated as (std dev)/sqrt(n) wouldn't be "statistically valid". Can you clarify? $\endgroup$ – Ben Bolker Dec 28 '18 at 21:13
  • 1
    $\begingroup$ By the law of large numbers, the mean is a consistent estimator under relatively weak assumptions. As such, its standard deviation (aka standard error) will approach 0 for large n. There is nothing wrong with the formula for large n. $\endgroup$ – Michael M Dec 28 '18 at 21:35
  • 4
    $\begingroup$ to follow up: why do you think it's artificially low? Can you tell us what it is about the small value of the standard error that's problematic for you? $\endgroup$ – Ben Bolker Dec 28 '18 at 21:40
  • 1
    $\begingroup$ What is it you think the standard error measures? $\endgroup$ – Glen_b -Reinstate Monica Dec 29 '18 at 10:05
  • 1
    $\begingroup$ There is nothing artificial about it being low. If you want to simulate a smaller data set, then just simulate fewer values. $\endgroup$ – Peter Flom - Reinstate Monica Dec 29 '18 at 12:28
1
$\begingroup$

Since you have a large sample (>2M), by the Strong Law of Large Numbers, the sample variance will converge to the population variance almost surely. See https://math.stackexchange.com/questions/243348/sample-variance-converge-almost-surely

The standard error is a good estimate of the population variance when you have small samples. The standard error is the variance of the sample mean in the sampling distribution.

$\endgroup$
  • 1
    $\begingroup$ I'm not convinced by your second paragraph. (1) what do you mean by "significant"? (2) why should the second sentence only be true for small samples? $\endgroup$ – Ben Bolker Dec 28 '18 at 21:51
  • $\begingroup$ My bad. Made a few edits. Applies for all sample sizes, as you rightly pointed out. $\endgroup$ – rgk Dec 28 '18 at 22:01
  • $\begingroup$ Standard error is $\sqrt{\dfrac{Var}{n}}$, and as such is a biased estimate of error, worse for small $n$. It is not a variance, per se. $\endgroup$ – Carl Dec 29 '18 at 1:35

Not the answer you're looking for? Browse other questions tagged or ask your own question.