Interest measure for frequent itemsets in databases of different size

Question

Consider a market basket analysis:

1. Among a certain number of transactions two items sometimes occur together. For simplicity let's assume they always occur together, but not in all transactions.
2. Now these two items occur together exactly the same number of times and again always together, but among much larger number of transactions. When I apply the common interest measure Lift, the lift in case #2 will be much higher than the lift in case #1, due to the fact that each item individually occurs more rarely.

But this does not feel intuitive. The two items occur more rarely, but their co-occurrence is the same.

An example would be frequent itemsets of Christmas-related items, when looking at grocery store transactions during December compared to a whole year.

I could counteract by somehow incorporating the Confidence. In fact the confidence relationship item#1 => item#2 and vice versa would be 100% for both cases presented above.

Which leads me to my question: Can you point me to any interest measures that incorporates confidence but also accounts for the number of occurences? Or solves in some other way my dilemma described above.

A link to a good paper would be most appreciated but everything, from a good online tutorial to a youtube video, helps.

Answer 1

You can consider normalized pointwise mutual information. In plain English, you compute a metric such that numerator represents the probability of a pair occurring together (using actual transactions data) whereas denominator represents probability of the pair occurring together by chance (probability of each item in the dataset). If a pair of products are more likely to occur together than by chance, the numerator will dominate the denominator. Essentially, we are trying to look for cases where the support for that pair in the dataset is not very big, but correlation is very significant.

This paper explains the usage of pointwise mutual information and its advantages in applications of item-set mining, what problems it solves etc.