Similarity measure between numbers I have a huge matrix (10*10k). I'd like to know if there is a way to find similarities between lines.
Let's give an example of matrix: 4*5
col1  col2  col3  col4
0     0     1     0
2     3     4     5
2     3     2     3
0     0.1   1     0
0     0     1     0

I'd like to know if there is a statistical theory to determine the similaritieis between data.
Line 1 is 100% like Line 5 Line 2 is 50% like Line 3
But how can I deal with numbers that are near to each others.
Line 4 and Line 5 have -nearly- same values. same how can we find a way to calculate probability of similarities ?
 A: Let's simplify this task a little bit, and let's assume, that the goal is to count how many values in two rows differ by no more than max_diff.
So, for a following matrix:
     a    b  c    d
0  0.0  0.5  1  0.5
1  2.5  3.0  4  5.0
2  2.0  3.0  3  4.5
3  2.0  3.0  1  0.0
4  0.0  0.0  1  0.5

and max_diff = 0.5 the resulting matrix of similarity of rows looks like this:
     0     1    2    3   4
0  NaN   NaN  NaN  NaN NaN
1  0.0   NaN  NaN  NaN NaN
2  0.0  0.75  NaN  NaN NaN
3  0.5  0.50  0.5  NaN NaN
4  1.0  0.00  0.0  0.5 NaN

[5 rows x 5 columns]

Now we can read it like:


*

*100% of values in the first row (0 in column) and the fourth row (4 in index) are similar or identical;

*50% of values in the first row (0 in column) are similar or identical to values in the fourth row (3 in index);

*75% of values in the second row (1 in column) are similar or identical to values in the third row (2 in index).


So, these two rows: [2.5, 3.0, 4, 5.0] and [2.0, 3.0, 3, 4.5] are 75% similar, because: abs(2.5 - 2.0) <= 0.5; and abs(3.0 - 3.0) <= 0.5; and abs(5.0 - 4.5) <= 0.5.
The lacking 25% are reserved for the row in which abs(4 - 3) > 0.5.
Here's the code:
import numpy as np
import pandas as pd

def count_similarity(row1, row2, max_diff=0.5):
   diff_array = abs(row1 - row2)
   no_sim_vals = sum(diff_array <= max_diff)
   return no_sim_vals / float(len(row1))

def find_similar_rows(df):
   res = pd.DataFrame(np.zeros((df.shape[0],)*2))
   res[:] = np.NAN

   for index1, row1 in df.iterrows():
      lrow1 = row1.values
      for index2, row2 in df.iterrows():
         if index1 == index2:
            res.ix[index1, index2] = np.NAN
            continue
         if index2 >= index1:
            break
         lrow2 = row2.values
         res.ix[index1, index2] = count_similarity(row1.values, row2.values)

   return res

if __name__ == '__main__':
   df = pd.DataFrame({ 'a': [0.0, 2.5, 2.0, 2.0, 0.0],
                       'b': [0.5, 3.0, 3.0, 3.0, 0.0],
                       'c': [1.0, 4.0, 3.0, 1.0, 1.0],
                       'd': [0.5, 5.0, 4.5, 0.0, 0.5], })
   print df
   print find_similar_rows(df)

