2
$\begingroup$

Lift is a measure widely used in many domains. However, it is known to have a problem for infrequent counts. What are the solutions for this type of problem?

In frequent pattern mining hyper-lift was suggested by Hahsler and Hornik (http://arxiv.org/pdf/0803.0966v1.pdf), but after some search it seems, it is either not used in other domains (or it is just used under other names). Moreover, it is almost the only article that discusses it (apart from R package arules). What is the reason? Is this measure reliable? What is a common used adjustment of the lift that is not affected by the problem of infrequency (becomes too volatile for smaller counts)?

$\endgroup$
2
$\begingroup$

Lift is a measure of correlation in a 2x2 contingency table. It tries to asses how much the occurrence of one item "lifts" the other. Given item A and B:

$$ Lift = \frac{P(A\cup B)}{P(A)P(B)} = \frac{confidence(A\rightarrow B)}{support(B)} $$

As mentioned, Lift can have problems. One problem is if the counts are imbalanced. In Han, Kamber, Pei (2012), they describe 3 other metrics to be considered in pattern matching:

$$ \begin{align} allConfidence(A,B) &= min \big\{P(A|B), P(B|A)\big\} = \frac{support(A\cup B)}{max\big\{support(A), support(B)\big\}}\\ maxConfidence(A,B) &= max \big\{P(A|B), P(B|A)\big\}\\ Kulczynski(A,B) &= \frac{1}{2}\big(P(A|B) + P(B|A) \big)\\ \end{align} $$

They all range from [0, 1] where positive association is 1, negative association is 0, and neutral association is 0.5. Here is a positive association example.

          A    not A       lift   allConf    maxConf    Kulc.
B      10000    1000       9.26      0.91      0.91     0.91
not B   1000  100000

Here is a negative association example.

          A    not A       lift   allConf    maxConf    Kulc.
B        100    1000       8.44      0.09      0.09     0.09
not B   1000  100000

Here is a neutral association example.

          A    not A       lift   allConf    maxConf    Kulc.
B       1000    1000       5.75       0.5        0.5     0.5
not B   1000  100000

The three new metrics perform well under the unbalanced data sets. They differ in some other data sets. One example is:

          A    not A       lift   allConf    maxConf    Kulc.
B       1000     100       9.18      0.09      0.91      0.5
not B  10000  100000

From one point of view, only $\frac{AB}{AB + A\bar{B}}=\frac{1000}{1000+10000}=9.09\%$ of A transactions contain B transactions, indicating negative assocation. Conversely, $\frac{AB}{AB +\bar{A}B}=\frac{1000}{1000+100}=90.9\%$ of B transactions contain A transactions, indicating a postive assocation. So depending on your needs in this situation, one of these three metrics should cover it.

The reference for this post is Section 6.3 of the text Data Mining, Concepts and Techniques 3rd ed. by Han, Kamber, Pei.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.