Finding similarity between two datasets [closed]

Question

I have two datasets. One is actual percentage of white population in counties in an american state and the other is the simulated percentage of white population in counties in an american state.

Bits about my simulation:

It is a random simulation done on California map with two different agents, white and minority. Their total population is based on the real white to minority ratio in California. For example if there is 70% white and 30% minority in California then the agents(say total 100) would be 70 white and 30 minority. First the map is randomly populated with both the agents and then around 100 iteration of the simulation is performed. On every iteration agent moves based on certain conditions. Data is taken after the 100th iteration and it includes what percentage of the white and minority are in a certain county in California.

So below is the data from the state of California

california_actual_white = [0.52, 0.72, 0.9, 0.86, 0.91, 0.91, 0.67, 0.79, 0.89, 0.77, 0.89, 0.84, 0.9, 0.81, 0.82, 0.81, 0.87, 0.82, 0.71, 0.86, 0.86, 0.9, 0.86, 0.82, 0.89, 0.91, 0.82, 0.84, 0.93, 0.72, 0.85, 0.91, 0.8, 0.64, 0.88, 0.77, 0.76, 0.54, 0.67, 0.89, 0.61, 0.85, 0.55, 0.87, 0.88, 0.94, 0.87, 0.61, 0.87, 0.83, 0.73, 0.9, 0.88, 0.88, 0.9, 0.84, 0.75, 0.79]

california_simulated_white = [0.48, 0.54, 0.6, 0.62, 0.66, 0.69, 0.71, 0.71, 0.71, 0.72, 0.74, 0.75, 0.77, 0.78, 0.79, 0.79, 0.8, 0.8, 0.8, 0.81, 0.81, 0.82, 0.82, 0.82, 0.83, 0.84, 0.85, 0.85, 0.87, 0.87, 0.87, 0.88, 0.91, 0.92, 0.93, 0.93, 0.94, 0.94, 0.94, 0.94, 0.95, 0.95, 0.97, 0.97, 0.98, 0.98, 0.98, 0.98, 0.99, 0.99, 0.99, 0.99, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]

How would I found a metric of similarity between these two datasets.

I found out a these three option can be used to find similarity and also all of them have a method in Python:

1) Earth mover's distance

2) Kullback–Leibler divergence

3) Cosine Similarity

But I have some doubts using these methods. They are

1) With Kullback-Leibler divergence and Cosine Similarity, the value changes if i reshuffle both the arrays and compute both the metrics again but with Earth movers distance its not the case. It will give you the same value for the two datasets regarding the reshuffling/position of the data points which made me think to use this metric.

2) But the second doubt is that you can use the K-L divergence and Earth movers distance only with two probability distribution. So i am not sure here that if the two datasets above are the probability distribution or not.

So my final two questions are -

1) Are the above datasets considered as probability distribution? If yes why?

2) If 1) is yes then what is the best method to determine the similarity. If 1) is no then what is the best method to determine the similarity ?

Before I was leaning towards Earth Movers Distance as I stated the reason in the doubts section.

Answer 1

1) almost so. The proportions can be considered as conditional probability of finding a white person, when looking for someone randomnly in each state. You may then define a joint probability function for the varibles "county" and "skin color". This means that, if you want to turn your list of proportions in a unique distribution, you have to include proportions of habitants of each county, so that your data will look a lot more like:

      | 1st county | 2nd county | 3rd county
white |         .2           .1           .2
black |         .2           .2           .1

Note that totals by columns are not equal, and that overall total is 1. You may also put all these probabilities in one vector, before processing them. If you fix the counties populations in your simulation to be the real ones, the similarity measure using this distributions will reflect only the differences between proportions of white people, giving more weight to more populous counties. If you don't want this, you may just consider each of your proportion as a probability value of a Bernoulli distribution, and comupute 58 similarities of those distribution, before averaging them.

2) it depends. There are infinite possibilities, and most of them are probably just fine. First ones that come to my mind are Chi-squared statistic and Cramer's phi, but also Kullback–Leibler divergence makes some sense here. Although you have to consider that KL divergence consideres one distribution to be the true one, and the other one to be an approximation of it. So if you consider your simulation as a kind of model to describe how people of different races distributes, you may want to use KL divergence (as well as chi squared or Cramer's phi) to describe how near your model gets to real data. But I would rather use an average of multiple simulations, or some estimate of expected outcome of your model. I can't understand now how your method work, but if you expect convergence in your simulation, then KL probably makes sense.

Histogram intersection is one other feasible method.

Cosine similarity is a less theorically founded similarity measure, not that much related to probability at all. You may want to use it as well as MSE or countless others. Cosine similarity works in a way that is usefull in other kinds of applications in particular and I don't see any reason why you should choose this measure over others, but still it could work just fine.

About EMD, you wrote:

1) With Kullback-Leibler divergence and Cosine Similarity, the value changes if i reshuffle both the arrays and compute both the metrics again but with Earth movers distance its not the case.

but this is just wrong, it is rather the opposite. If you shuffle your data (provided that you shuffle both your vectors of probabilities keeping them paired) all the methods I've discussed since now give the same results, you can verify it by looking into their formulas. EMD instead needs (and it is the only one since now) a metric to evaluate distance between various counties. Of course actual distance between, let's say, the capitals of the counties, immediately comes to mind. So, if you want to take into account the geographical position of data, you can choose this metric (and you need to use their coordinates of course). If your simulation doesn't take into account geographical distances, I would rather not use it.