I have a large (N=10^6) Monte Carlo ensemble of runs of my engineering code. I want to compute a confidence interval for the means of two output variables, and falsify the hypothesis that they have the same population mean. The two variables are not normally distributed (they do have finite variances, though). Given the very large number of runs, I believe the sample means are normally distributed. To check this assumption, I'm thinking to divide the Monte Carlo ensemble in m=10^3 smaller ensembles, and compute the corresponding 10^3 samples means. I could then make a simple QQ plot of the distribution of these sample means, and see if it's approximately normal. If this is true, then a fortiori I expect the normality approximation to hold for the sample mean of the full Monte Carlo ensemble. Is this correct? After all, my samples are i.i.d., thus I can divide the full Monte Carlo ensemble in as many disjoint sets as I want, average the samples in each set, and still get i.i.d variables.
Some quick points:
- It sounds a bit like your two output variables could be paired. This is the case if you get one value for each variable per run. In this case normality of the individual variables is not important anyway - you would look at the distribution of the pairwise differences instead.
- Given a sample size of N = 10^6 unless your data is really crazy normality is a non-issue.
- I would look at the bootstrap distribution instead since it is more common and more efficient than the procedure you describe.
- Are you certain about iid or did you look at it? Some issues I observed in the past: CPU downclocking due to heat. Autocorrelation between subsequents runs due to memory allocation.