0
$\begingroup$

I used cosine angle to characterize the similarity of two vectors $x_{1}$ and $x_{2}$, and then performed permutation test to evaluate the significance of the similarity. For permutation test, vector $x_1$ was randomized for 10,000 times and recalculate the similarity between randomized $x_1$ and $x_2$ as null similarity scores. Generally in my research field, two vectors with similarity score higher than 0.7 are considered that the two vectors are similar, and p values are always lower than 0.01 in my experiments. But I encountered a problem, there are some vector pairs, the similarity scores are as lower as 0.35, which are always not acceptable as similar vector pairs. However, the permutation tests showed that they are significant. How can I handle these data? Accept the permutation test results that the two vectors are similar, or reject the comparison that the two vectors are dissimilar?

As an example, I used the following Python codes to perform similarity score calculation and permutation test. For calculating p value, Weibull distribution is fitted to the distribution of null similarity scores using scipy's weibull_min function.

import math, random
from scipy import stats
similarity = lambda x1, x2: sum(xj*xk for xj,xk in zip(x1, x2))/math.sqrt(sum(xj**2 for xj in x1)*sum(xk**2 for xk in x2))
x1 = [0.0032569599818743095, 0.0018352261463083067, 0.1544195530884477, 0.01599592171967487, 0.010042765300631567, 0.003948001925854598, 0.023976889744824266, 0.023477769407233284, 0.002962417513948285, 0.40116117703701604, 0.003942337647625251, 0.04431732419496446, 0.16025943527146053, 0.005216800249228242, 0.0052791073097510546, 0.3664334872128919, 0.01542382961851086, 0.0056699425075759724, 0.0060381205924835025, 0.02871789062278739, 0.0056586139511172785, 0.05169544988529837, 0.014307966807329577, 0.012166869636636551, 0.0033305955988558156, 0.026927978702313858, 0.03333427737970489, 0.00501855051120111, 0.010082415248236993, 0.00790733240816789, 0.03490894672746325, 0.08398991758475176, 0.003687445127304653, 0.0028774533405080856, 0.002429316604831629, 0.035911523974057606, 0.010473250446061911, 0.0036704522926166135, 0.006978390778555043, 0.0030643745220765243, 0.0025602537596646747, 0.008660681412670991, 0.004639043869834886, 0.0024866181426831686, 0.012535047721544082, 0.003914016256478518, 0.002073125831940865, 0.0026508822113342208, 0.002679203602480954, 0.00364213090146988, 0.0022203970659038772, 0.0034552097199014417, 0.002871789062278739, 0.0019598402673539324, 0.006593219858959472, 0.004825965051403325, 0.003993316151689371, 0.016624656603132344, 0.006332663060409528, 0.033985669376079754, 0.005024214789430457, 0.0027301821065450734, 0.006049449148942196, 0.002537596646747288, 0.018408904245376532, 0.0447308051771503, 0.003721430796680733, 0.018924353564247078, 0.007873346738791809, 0.004423801297119715, 0.12999518536350504, 0.001865999603500524, 0.013384689455946076, 0.02888781896966779, 0.08400124614121046, 0.028491319493613526, 0.0021864113965277977, 1.0, 0.006712169701775751, 0.013095811266249398, 0.0778838256535161, 0.19062561953043133, 0.0022600470135093038, 0.06422725084256138, 0.008088589311506982, 0.0036145271743748056, 0.005403721430796681, 0.0038743663088730918, 0.00364213090146988, 0.009159137896853494, 0.15238607720411226, 0.05696564615253901, 0.0044747798011838345, 0.0018862046503724263, 0.00459939392222946, 0.1362088985810983, 0.043898156277436345, 0.0068877623268854965, 0.004282194341386049, 0.0023846611345549294, 0.0016746313970942253, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
x2 = [0.030350932658868162, 0.031262282643060385, 0.06896171356307303, 0.04452535567499209, 0.09864053114132153, 0.0445036990199178, 0.27189377173569396, 0.27189377173569396, 0.027821688270629148, 0.9797660448940879, 0.03260461587100853, 0.09389819791337338, 0.39709136895352515, 0.05575735061650332, 0.04521024343977237, 0.05659212772684161, 0.12804299715460007, 0.05469490989566867, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.03509326588681631, 0.0366745810938982, 0.028137843819159025, 0.2671514385077458, 0.030034777110338286, 0.07714195384128991, 0.02845428390768258, 0.04370907366424281, 0.028453999367688904, 0.028453999367688904]
s = similarity(x1, x2)
## permutation test
lx, sr = len(x1), []
for j in xrange(10000):
    mj = random.sample(x1, lx)
    sr.append(similarity(mj, x2))
shape, loc, scale = stats.weibull_min.fit(sr)
## -log10(p)
ej = ((s-loc)/scale)**shape*math.log10(math.exp(1.))
p = 10**(-ej)

The results are:

s = 0.35210496894867593 # similarity score
p = 0.0071377151009145226 # permutation test p value
$\endgroup$
0
$\begingroup$

A small p-value would just indicate that the 0.352 is a statistically significant result (that you are not getting that cosine similarity number just by chance) but you still have to decide whether 0.35 is acceptable or too low based on your experience / acceptability in your field

Just as an example, if your x1 and x2 vectors are too short, you can get a similar cosine similarity number but the p-value is high, indicating that the cosine similarity number could just be a chance occurence

I dont use Python, so I am giving an example in R, but you can just use the shortened x1, x2 vectors in your python code

x1 = c(0.0032569599818743095, 0.0018352261463083067, 0.1544195530884477) 
x2 = c(0.030350932658868162, 0.031262282643060385, 0.01596171356307303)

cs = (x1%*%x2) / ( sqrt(sum(x1^2)) * sqrt(sum(x2^2)) )

fn <- function(x) {
xtemp = sample(x1, length(x1) )
return((xtemp%*%x2) / ( sqrt(sum(xtemp^2)) * sqrt(sum(x2^2)) )  )
}

tmp = sapply(1:10000,fn)
mean( abs(tmp) > as.numeric(cs) )

Output:
cs = 0.366
p-value = 0.836
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.