I have a large (N=10^6) Monte Carlo ensemble of runs of my engineering code. I want to compute a confidence interval for the means of two output variables, and falsify the hypothesis that they have the same population mean. The two variables are not normally distributed (they do have finite variances, though). Given the very large number of runs, I believe the sample means are normally distributed. To check this assumption, I'm thinking to divide the Monte Carlo ensemble in m=10^3 smaller ensembles, and compute the corresponding 10^3 samples means. I could then make a simple QQ plot of the distribution of these sample means, and see if it's approximately normal. If this is true, then a fortiori I expect the normality approximation to hold for the sample mean of the full Monte Carlo ensemble. Is this correct? After all, my samples are i.i.d., thus I can divide the full Monte Carlo ensemble in as many disjoint sets as I want, average the samples in each set, and still get i.i.d variables.
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ Yes, that could work. $\endgroup$– kjetil b halvorsen ♦Commented Sep 29, 2015 at 10:19
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
Some quick points:
- It sounds a bit like your two output variables could be paired. This is the case if you get one value for each variable per run. In this case normality of the individual variables is not important anyway - you would look at the distribution of the pairwise differences instead.
- Given a sample size of N = 10^6 unless your data is really crazy normality is a non-issue.
- I would look at the bootstrap distribution instead since it is more common and more efficient than the procedure you describe.
- Are you certain about iid or did you look at it? Some issues I observed in the past: CPU downclocking due to heat. Autocorrelation between subsequents runs due to memory allocation.
-
$\begingroup$ Hi, Erik, thanks for the answers! 1. Good point! $Y_1$ and $Y_2$ are indeed paired. Do you think $Y_1-Y_2$ may be normal, even if each one is not? I will check. 2. :) I agree, but you know...there are always nitpickers (a guy at a conference was asking for 10^8 samples!!!!) 3. I will definitely use boostrap from now on - I know about it, but I always forget to use it! $\endgroup$– DeltaIVCommented Sep 29, 2015 at 14:35
-
1$\begingroup$ 4. I didn't verify that the sequence is i.i.d. I'm not even sure how I would do that. However, I'm using default R rng (Mersenne-Twister). It's touted as having "A twisted GFSR with period 2^19937 - 1 and equidistribution in 623 consecutive dimensions (over the whole period)". The period is very large with respect to 10^6, so I hope it's good enough! Also, as far as my limited understanding of random number generators goes, it should be a deterministic random number generator, so I don't understand why CPU temperature or memory allocation would have an influence. $\endgroup$– DeltaIVCommented Sep 29, 2015 at 14:38
$\begingroup$
$\endgroup$
I think this question's answer answers your question: Techniques/diagnostics for gaining confidence in normality assumptions and resulting confidence intervals
which is what I think @Erik's bullet 3 was suggesting.
