0
$\begingroup$

I have a very large dataset (with > 2 million simulated values). I want to compute standard error for this dataset. To do that, I divide the standard deviation by square root of number of observations. However, because of the large number of observations, the standard error is quite low. Is there a way to subsample instead and compute standard error?

$\endgroup$

closed as unclear what you're asking by Ben Bolker, kjetil b halvorsen, Peter Flom Dec 29 '18 at 12:29

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    $\begingroup$ At first glance, it's not obvious why the standard error calculated as (std dev)/sqrt(n) wouldn't be "statistically valid". Can you clarify? $\endgroup$ – Ben Bolker Dec 28 '18 at 21:13
  • 1
    $\begingroup$ By the law of large numbers, the mean is a consistent estimator under relatively weak assumptions. As such, its standard deviation (aka standard error) will approach 0 for large n. There is nothing wrong with the formula for large n. $\endgroup$ – Michael M Dec 28 '18 at 21:35
  • 4
    $\begingroup$ to follow up: why do you think it's artificially low? Can you tell us what it is about the small value of the standard error that's problematic for you? $\endgroup$ – Ben Bolker Dec 28 '18 at 21:40
  • 1
    $\begingroup$ What is it you think the standard error measures? $\endgroup$ – Glen_b Dec 29 '18 at 10:05
  • 1
    $\begingroup$ There is nothing artificial about it being low. If you want to simulate a smaller data set, then just simulate fewer values. $\endgroup$ – Peter Flom Dec 29 '18 at 12:28
1
$\begingroup$

Since you have a large sample (>2M), by the Strong Law of Large Numbers, the sample variance will converge to the population variance almost surely. See https://math.stackexchange.com/questions/243348/sample-variance-converge-almost-surely

The standard error is a good estimate of the population variance when you have small samples. The standard error is the variance of the sample mean in the sampling distribution.

$\endgroup$
  • 1
    $\begingroup$ I'm not convinced by your second paragraph. (1) what do you mean by "significant"? (2) why should the second sentence only be true for small samples? $\endgroup$ – Ben Bolker Dec 28 '18 at 21:51
  • $\begingroup$ My bad. Made a few edits. Applies for all sample sizes, as you rightly pointed out. $\endgroup$ – rgk Dec 28 '18 at 22:01
  • $\begingroup$ Standard error is $\sqrt{\dfrac{Var}{n}}$, and as such is a biased estimate of error, worse for small $n$. It is not a variance, per se. $\endgroup$ – Carl Dec 29 '18 at 1:35

Not the answer you're looking for? Browse other questions tagged or ask your own question.