I have a list of words, with two statistics computed for each entry, i.e.

Word           Freq   MI
beautiful girl 2310   12.07
girl gift       50    14.9

2310 is the frequency of 'beautiful girl' in a corpus and 12.07 is the mutual information computed of the two words. In order to guess which phrase "beautiful girl" or "girl gift" should be kept and the other one is thus discarded, I want a single metric computed based on Freq and MI to measure the association strength of two words in each phrase.

The simplest way to get a single measure is to compute a simple average score:

(Freq + MI)/2

But this may not be the best way to do it. The value range of Freq is [1, 3000] and the value range MI is [-7.2222, 14.0999] in my data.

What's the sensible way to compute a single metric to compare? It has to be noted that the the weight of MI should be much more higher than Freq. In that case, I may need a constant to multiply the MI first, and then calculate the average. That can effectively downplay the importance of Freq, compared to MI.


Perhaps, it could be a solution if you first transform the values of beautiful girl and girl gift in a range between zero and one. After this you could calculate the mean.

  • $\begingroup$ Does this help to compress the large freq more than the small freq? I want an effect of normalization that reduces the large freq more than those small freqs. The thinking is similar to the idea of regularization parameter typically used in machine learning optimization. The freq here can be considered as 'weight' of mutual information. $\endgroup$ – user697911 Oct 13 '17 at 20:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.