3
$\begingroup$

I'm wondering if there is a possibility to perform hypothesis tests to show that the resulting rules from an apriori market basket analysis are statistically significant?

Maybe I'm missing the forest through the trees so any kind of help or hint is highly appreciated.

$\endgroup$
2
  • 1
    $\begingroup$ Statistical significance and correctness are completely different things: could you clarify what you are trying to accomplish? $\endgroup$
    – whuber
    Jan 9, 2020 at 19:20
  • $\begingroup$ Thank you for pointing this out. I'd like to show thr statistical significance of the results. $\endgroup$ Jan 9, 2020 at 20:15

1 Answer 1

3
$\begingroup$

The problem with Frequent Pattern Mining is that, in practice, one is likely to discover a titanic number of patterns with many redundancies. Many researchers have devoted substantial effort to come up with clever ways to improve the quality of pattern discovery, i.e. bring forth the most interesting patterns and bring down redundancy.

The word 'interesting' can be formalised in more than one way depending on the problem but a high-level example of a formalisation could be to take the word 'interesting' to qualify a pattern that strongly deviates on its expected support under some user-defined distributional null hypothesis.

The Independence Model is the simplest distributional model : one assumes that all individual items appear independently. So, the expected frequency of a pattern $X=\{x_1,...,x_n\}$ equals $\prod_{i=1}^{n} fr(x_i)$, where the frequencies $fr(x_i)$ are simply computed from the historical data.

Presently, we can calculate both $fr(X)$, the observed frequency, and the expected frequency $ind(X)$ of the independence model. The ratio $$ \frac{fr(X)}{ind(X)}$$ is called the lift of $X$. Suppose we consider patterns that have a high lift to be interesting. Now, suppose our dataset contains $N$ transactions and the transactions are independent.

  • According to the independence model, the probability of generating a transaction that contains a pattern $X$ equals $fr(X)$;
  • The probability $$ P(X\hspace{1mm}occurs\hspace{1mm}in\hspace{1mm}M\hspace{1mm}transactions) = {N\choose M} q^M (1-q)^{N-M},$$ where $q=ind(X)$.

Let $Z_X$ be the binomial random variable that counts the support of $X$ under the independence model.

Now, one can perform a one-sided statistical test by calculating $P(Z_X \geq Nfr(X))$.

Because $M$ is large, one can approximate the binomial by the normal distribution $$\mathcal{N}(\mu=Nq, \sigma^2=Nq(1-q)),$$ and ultimately obtain a good approximation of the p-value of the null hypothesis of the one-sided test by calculating the probability mass of the tail of a Gaussian distribution provided that the number of transactions is very large. The most interesting patterns would be the ones that lead to the lowest p-values.

Other relevant approaches are dealt with in Chapter $5$ of Frequent Pattern Mining.

$\endgroup$
2
  • $\begingroup$ Thank you for your detailed answer. $\endgroup$ Jan 13, 2020 at 14:10
  • $\begingroup$ You are most welcome. $\endgroup$ Jan 14, 2020 at 14:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.