I want to compare two lists with different lengths. There should be a percentage for their similarity as result. An example:
List 1: main ID: 233423 ID: no. 1324: 312 22433: 259 244: 5236 List 2: main ID: 2831 ID: no. 1324: 8687 231: 9283 244: 4872
In context, this means: the main ID 233423 occurs together 312 times with ID 1324, 259 times with ID 22433 and so on. The same for the second list.
Now I want to compare the lists of two different main IDs for their similarity.
IDs which are in both lists obviously should have a positive impact on the similarity. But because the absolute numbers can be totally different, I already compare the share of the total volume, so if ID 1324 is in both lists, I compare this share like
share1324List1 = 312/sumOfNoList1 share1324List2 = 8687/sumOfNoList2 |share1324List1-share1324List2| = simil1324
Then, I sum these values up and calculate the average, let's call it
avgAll = (siml1324+suml244)/2
But only comparing IDs which are in both lists and ignoring all other entries which are not in both lists is far too inaccurate, so the IDs which are only in one list need to be included somehow (with a negative impact on the similarity). I'm not sure how to do this in a nice way.
At the moment I sum up the no. values of entries which are in both lists and look how their share of the total amount is and simply multiply:
(if share of "volume" that is in both lists is bigger, the average similarity is reduced less...)
Is there a better way to compare those two lists? Or, better, is there any "best practice"? I've already seen the Kolmogorov-Smirnov test, but it seems to be not appropriate, because the ID itself means nothing, so it's not some x for the probability distribution - you can't sort by that ID and say the functions now should look the same. Any ideas?