How to compare different distributions with reference truth value in Matlab?

Question

I have production (q) values from 4 different methods stored in the 4 matrices. Each of the 4 matrices contains q values from a different method as:

Matrix_1 = 1 row x 20 column 

Matrix_2 = 100 rows x 20 columns 

Matrix_3 = 100 rows x 20 columns 

Matrix_4 = 100 rows x 20 columns 

The number of columns indicate the number of years. 1 row would contain the production values corresponding to the 20 years. Other 99 rows for matrix 2, 3 and 4 are just the different realizations (or simulation runs). So basically the other 99 rows for matrix 2,3 and 4 are repeat cases (but not with exact values because of random numbers).

Consider Matrix_1 as the reference truth (or base case ). Now I want to compare the other 3 matrices with Matrix_1 to see which one among those three matrices (each with 100 repeats) compares best, or closely imitates, with Matrix_1.

How can this be done in Matlab?

I know, manually, that we use confidence interval (CI) by plotting the mean of Matrix_1, and drawing each distribution of mean of Matrix_2, mean of Matrix_3 and mean of Matrix_4. The largest CI among matrix 2, 3 and 4 which contains the reference truth (or mean of Matrix_1) will be the answer.

mean of Matrix_1 = (1 row x 1 column)

mean of Matrix_2 = (100 rows x 1 column)

mean of Matrix_3 = (100 rows x 1 column)

mean of Matrix_4 = (100 rows x 1 column)

I hope the question is clear and relevant to SA. Otherwise please feel free to edit/suggest anything in question. Thanks!

Answer 1

It's not clear why you must have a confidence interval. As @whuber pointed out, there are better ways to compare distributions. You are losing some information by looking only at the mean. However, if you must, you might want to compute a confidence interval on k1_mean based on the 40000 observations. See Wikipedia for the simple explanation of how to compute this. From there, you could just count the number of the 100 values of k2_mean that fall into a confidence interval on k1_mean, and repeat for the third and fourth methods as well. Again, this isn't the best way to compare distributions.

Answer 2

This is two questions; (1) has good standard explanations available anywhere, so I'll address (2). You can compare distributions using a Kolmogorov-Smirnov statistic: it is the maximum difference between their empirical distribution functions. (The value of the edf at a number $x$ is the proportion of data less than or equal to $x$.) Its distribution depends on the base distribution and on the size of the sample distribution, so in this case about the only option is to find the distribution of the KS statistic through simulation: draw several thousand (at least) samples of size 100 from the reference distribution in a way that emulates how batches B, C, and D were originally created. Compute the KS statistic for each sample: this is your simulation, which you can summarize with a histogram. Also compute the KS statistic for B, C, and D. Where they are situated with respect to the simulation results informs you about how close B, C, and D are to the reference.

NB: This answer responds to the question as originally posed: "1) Explain the concept of CI in brief. 2) How do I use CI to find which one among B, C or D imitates or is close to the A?"