0
$\begingroup$

I found a python code on GitHub and I'm trying to understand it for using it on my project.

I am studying tweets collected from twitter, and I have the evolution of the frequency appearance of different words over time. At some point, I want to check the correlation coefficient of two words to know if they are likely to be part of the same tweet.

On the code I found on GitHub they use Erdem Correlation (that's the name they used for the method) to get that coefficient. The code they use is the following one:

def erdem_correlation(array_1, array_2):
    a_12 = 0.
    a_1 = 0.
    a_2 = 0.
    for i in range(1, len(array_1)):
        a_12 += (array_1[i] - array_1[i-1]) * (array_2[i] - array_2[i-1])
        a_1 += (array_1[i] - array_1[i-1]) * (array_1[i] - array_1[i-1])
        a_2 += (array_2[i] - array_2[i-1]) * (array_2[i] - array_2[i-1])
    a_1 = np.sqrt(a_1/(len(array_1) - 1))
    a_2 = np.sqrt(a_2/(len(array_2) - 1))
    coefficient = a_12/(len(array_1) * a_1 * a_2)
    return coefficient

And then to get the weight they do this:

weight = (st.erdem_correlation(main_word_freq, candidate_word_freq) + 1) / 2

Where main_word_freq and candidate_word_freq are lists containing the frequency of words usage, each on every position of the list.

I would like to know more about this Erdem Correlation as I have tried to find it on internet and haven't find anything about it. Moreover, is this way of getting the correlation coefficient of two variables that changes over the time appropiate or is there any better way to get it?

I also think that the method has an error, since in the first lap when doing the array_*[i-1] it will compare the first value with the last, which has no sense as they are not contiguous values over the time. What would be the correct code for the first lap

EDITED:

GitHub repository here.

$\endgroup$
  • 1
    $\begingroup$ It might be helpful to include a reference to said Github repository $\endgroup$ – deemel Apr 29 '19 at 14:19
  • $\begingroup$ @Rickyfox Edited $\endgroup$ – Luiscri Apr 29 '19 at 16:31
0
$\begingroup$

I think what they implemented is the dependency measure proposed by Erdem et al. (2014), hence the naming. You can find the paper here. (Sorry that I post this as an answer! I would rather post it as a comment, but I'm still not eligible to do so.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.