1
$\begingroup$

I'm doing a market basket analysis of items purchases in particular stores. Each dataset contains over 5 million transactions. I'm currently processing this in R, but it seems my computer can't handle such a large data file.

I notice that there are a lot of single-item purchases. Is it valid to disregard single item purchases and focus only on baskets containing 2 or more items?

$\endgroup$
0
$\begingroup$

No, you can't discard transactions, if you want to calculate any measures to evaluate how strong or good rules are.

E.g. let your items be A, B, C and the transactions (baskets)

A
A,B
A,B
C

You can calculate absolute frequency fr(A,B)=2 without caring about single A and single C. However, you can't evaluate the strength of rule A->B. If you include also single A you'll get fr(A)=3 correctly. Now you can calculate precision ('confidence') of A->B, P(B|A)=2/3. However, you will also need C to get the data size (n) correctly, here n=4. Now you can get the relative frequencies P(A,B)=2/4=0.5, P(A)=0.75, P(B)=0.5. Leverage of A->B is P(A,B)-P(A)P(B)=0.5-0.75*0.5=0.125.

Note: The problem is computationally demanding, if you want to find optimal rules without excessive minimum frequency thresholds. I would recommend and efficient C or Java implementation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.