# Calculate standard error for very large number of observations [closed]

I have a very large dataset (with > 2 million simulated values). I want to compute standard error for this dataset. To do that, I divide the standard deviation by square root of number of observations. However, because of the large number of observations, the standard error is quite low. Is there a way to subsample instead and compute standard error?

• At first glance, it's not obvious why the standard error calculated as (std dev)/sqrt(n) wouldn't be "statistically valid". Can you clarify? – Ben Bolker Dec 28 '18 at 21:13
• By the law of large numbers, the mean is a consistent estimator under relatively weak assumptions. As such, its standard deviation (aka standard error) will approach 0 for large n. There is nothing wrong with the formula for large n. – Michael M Dec 28 '18 at 21:35
• to follow up: why do you think it's artificially low? Can you tell us what it is about the small value of the standard error that's problematic for you? – Ben Bolker Dec 28 '18 at 21:40
• What is it you think the standard error measures? – Glen_b Dec 29 '18 at 10:05
• There is nothing artificial about it being low. If you want to simulate a smaller data set, then just simulate fewer values. – Peter Flom Dec 29 '18 at 12:28

Since you have a large sample (>2M), by the Strong Law of Large Numbers, the sample variance will converge to the population variance almost surely. See https://math.stackexchange.com/questions/243348/sample-variance-converge-almost-surely

The standard error is a good estimate of the population variance when you have small samples. The standard error is the variance of the sample mean in the sampling distribution.

• I'm not convinced by your second paragraph. (1) what do you mean by "significant"? (2) why should the second sentence only be true for small samples? – Ben Bolker Dec 28 '18 at 21:51
• My bad. Made a few edits. Applies for all sample sizes, as you rightly pointed out. – rgk Dec 28 '18 at 22:01
• Standard error is $\sqrt{\dfrac{Var}{n}}$, and as such is a biased estimate of error, worse for small $n$. It is not a variance, per se. – Carl Dec 29 '18 at 1:35