# How to compare different distributions with reference truth value in Matlab?

I have production (q) values from 4 different methods stored in the 4 matrices. Each of the 4 matrices contains q values from a different method as:

Matrix_1 = 1 row x 20 column

Matrix_2 = 100 rows x 20 columns

Matrix_3 = 100 rows x 20 columns

Matrix_4 = 100 rows x 20 columns


The number of columns indicate the number of years. 1 row would contain the production values corresponding to the 20 years. Other 99 rows for matrix 2, 3 and 4 are just the different realizations (or simulation runs). So basically the other 99 rows for matrix 2,3 and 4 are repeat cases (but not with exact values because of random numbers).

Consider Matrix_1 as the reference truth (or base case ). Now I want to compare the other 3 matrices with Matrix_1 to see which one among those three matrices (each with 100 repeats) compares best, or closely imitates, with Matrix_1.

How can this be done in Matlab?

I know, manually, that we use confidence interval (CI) by plotting the mean of Matrix_1, and drawing each distribution of mean of Matrix_2, mean of Matrix_3 and mean of Matrix_4. The largest CI among matrix 2, 3 and 4 which contains the reference truth (or mean of Matrix_1) will be the answer.

mean of Matrix_1 = (1 row x 1 column)

mean of Matrix_2 = (100 rows x 1 column)

mean of Matrix_3 = (100 rows x 1 column)

mean of Matrix_4 = (100 rows x 1 column)


I hope the question is clear and relevant to SA. Otherwise please feel free to edit/suggest anything in question. Thanks!

-
No one to explain? –  Pupil Sep 14 '10 at 2:18
Do you really mean you have 1 observation from A? It seems like you are likely to make an error with so little data. –  shabbychef Sep 14 '10 at 3:17
Its not one observation, it is one distribution or realization with 40000 values. –  Pupil Sep 14 '10 at 3:38
So say a sample has size n = 40000. Then you have one such sample of random variates from A, and 100 replicates each for B, C and D? –  ars Sep 14 '10 at 3:55
@ars: Yes. However A,B,C,D are not necessarily related. –  Pupil Sep 14 '10 at 15:15
show 1 more comment

It's not clear why you must have a confidence interval. As @whuber pointed out, there are better ways to compare distributions. You are losing some information by looking only at the mean. However, if you must, you might want to compute a confidence interval on k1_mean based on the 40000 observations. See Wikipedia for the simple explanation of how to compute this. From there, you could just count the number of the 100 values of k2_mean that fall into a confidence interval on k1_mean, and repeat for the third and fourth methods as well. Again, this isn't the best way to compare distributions.

-
Regarding the wikipedia link that you gave which requires data to be normally distributed, however my data is not presumed to be normal or of any other distribution. Further, what is the best way to compare distributions if you think the mentioned method is not? Thanks ! –  Pupil Sep 25 '10 at 3:49
@Harpreet: true enough, however, if your distribution has a finite variance, the central limit theorem ( en.wikipedia.org/wiki/Central_limit_theorem ) describes the asymptotic distribution of the sample mean. Since you presumably do not know the population variance, you would have to estimate that as well. see also en.wikipedia.org/wiki/Student's_t-statistic#Prediction –  shabbychef Sep 25 '10 at 3:58
This is two questions; (1) has good standard explanations available anywhere, so I'll address (2). You can compare distributions using a Kolmogorov-Smirnov statistic: it is the maximum difference between their empirical distribution functions. (The value of the edf at a number $x$ is the proportion of data less than or equal to $x$.) Its distribution depends on the base distribution and on the size of the sample distribution, so in this case about the only option is to find the distribution of the KS statistic through simulation: draw several thousand (at least) samples of size 100 from the reference distribution in a way that emulates how batches B, C, and D were originally created. Compute the KS statistic for each sample: this is your simulation, which you can summarize with a histogram. Also compute the KS statistic for B, C, and D. Where they are situated with respect to the simulation results informs you about how close B, C, and D are to the reference.
NB: This answer responds to the question as originally posed: "1) Explain the concept of CI in brief. 2) How do I use CI to find which one among B, C or D imitates or is close to the A?"