Compare multi-histograms for goodness of fit

Question

I want to compare two distributions, however each of them can be shown as a group of histograms. These groups are basically has similar structures. What is the best way to compare them using goodness of fit methods?

My own idea is to compare each two histogram pairwise using known goodness of fit methods (for example chi-squared) and at the end calculate an average (or median, standard deviation etc.) of these differences.

To be more clear about the method, let me give you an example very similar to my data:

Suppose we want to compare pollution of two countries, our approach is to compare cities pairwise. For instance when comparing China with Italy, we compare four most populated cities pair-wise (numbers in parenthesis represent p-value in chi-squared test):

• Rome vs Shanghai (0.25)
• Milan vs Beijing (0.35)
• Naples vs Tianjin (0.85)
• Turin vs Guangzhou (0.50)

Because of average lower than 0.5 we may conclude Italy and China are not similar (0.25+0.35+0.85+0.50)/4=0.4875), however Italy and France that may result in 0.75 are much more similar.

Do you think this approach is correct? Is there any more accurate approach you are aware of?

Moreover, I would like to "weight" the p-value of goodlness of fit based on population.

PS:I am not a statistician! forgive my mistakes!!!

Answer 1

My guess is that you mix several things that cannot be connected like you assume. Actually there are several open questions you would need to answer, before a good answer could be given.

From your question I gather you have some distributions, currently in unparametrized form -- as a set of histograms, indicating that the distributions are multivariate, that is: more than one dimension. Which gives you the first keyword.

Then you would like to compare the individual histograms, and quantify how well they match. There are few tests that will give you a "scoring" answer. I further guess that you don't know in advance, which distribution is "right". The only thing I can think of with this little knowledge is a Kolmogorov-Smirnov test. In principle this will give you a hint, whether the histograms have been drawn from the same basic distribution. But beware what you make of this score, it is not to be interpreted as a probability. Also beware that the "resolution power" of the test is rather weak.

From my perspective you completely misplace "fitting" in this context. Basically fitting -- or parameter estimation in general -- means that you have a parametrized model. E.g. some polynomial expression of your random variable. Then you could find the best fitting parameters by a given procedure. e.g. least squares, likelihoods. Some of these procedures (!) will give you a goodness-of-fit information. Usually the quality measures are related by the harshness of requirements. A chi-square test will only work on a problem, where you have gaussian errors for your parameters. A likelihood method is far less requiring, but won't give you any hint at the goodness-of-fit.

And as a last point: if you have multivariate distribution, usually correlation exists. If you only use the projections you omit the non vanishing covariance, which results in serious errors when you start to extract/fit parameter values.

And let's say you have extracted the parameters including the errors for each distribution correctly. Then it is simply a matter of comparing the parameter values and their errors. E.g. if you have extracted the parameters $$ x_i = 0.5 \pm .012$$ and for another sample $$ x_j = 0.3 \pm .072$$ -- then you could state that there is a small probability that they match. In this context the errors are usually Gaussian, so you can actually look up how much overlap these results have.