I found a python code on GitHub and I'm trying to understand it for using it on my project.
I am studying tweets collected from twitter, and I have the evolution of the frequency appearance of different words over time. At some point, I want to check the correlation coefficient of two words to know if they are likely to be part of the same tweet.
On the code I found on GitHub they use
Erdem Correlation (that's the name they used for the method) to get that coefficient. The code they use is the following one:
def erdem_correlation(array_1, array_2): a_12 = 0. a_1 = 0. a_2 = 0. for i in range(1, len(array_1)): a_12 += (array_1[i] - array_1[i-1]) * (array_2[i] - array_2[i-1]) a_1 += (array_1[i] - array_1[i-1]) * (array_1[i] - array_1[i-1]) a_2 += (array_2[i] - array_2[i-1]) * (array_2[i] - array_2[i-1]) a_1 = np.sqrt(a_1/(len(array_1) - 1)) a_2 = np.sqrt(a_2/(len(array_2) - 1)) coefficient = a_12/(len(array_1) * a_1 * a_2) return coefficient
And then to get the weight they do this:
weight = (st.erdem_correlation(main_word_freq, candidate_word_freq) + 1) / 2
candidate_word_freq are lists containing the frequency of words usage, each on every position of the list.
I would like to know more about this Erdem Correlation as I have tried to find it on internet and haven't find anything about it. Moreover, is this way of getting the correlation coefficient of two variables that changes over the time appropiate or is there any better way to get it?
I also think that the method has an error, since in the first lap when doing the
array_*[i-1] it will compare the first value with the last, which has no sense as they are not contiguous values over the time. What would be the correct code for the first lap
GitHub repository here.