Hypothesis testing for apriori market basket analysis I'm wondering if there is a possibility to perform hypothesis tests to show that the resulting rules from an apriori market basket analysis are statistically significant?
Maybe I'm missing the forest through the trees so any kind of help or hint is highly appreciated.
 A: The problem with Frequent Pattern Mining is that, in practice, one is likely to discover a titanic number of patterns with many redundancies. Many researchers have devoted substantial effort to come up with clever ways to improve the quality of pattern discovery, i.e. bring forth the most interesting patterns and bring down redundancy. 
The word 'interesting' can be formalised in more than one way depending on the problem but a high-level example of a formalisation could be to take the word 'interesting' to qualify a pattern that strongly deviates on its expected support under some user-defined distributional null hypothesis.
The Independence Model is the simplest distributional model : one assumes that all individual items appear independently. So, the expected frequency of a pattern $X=\{x_1,...,x_n\}$ equals $\prod_{i=1}^{n} fr(x_i)$, where the frequencies $fr(x_i)$ are simply computed from the historical data.
Presently, we can calculate both $fr(X)$, the observed frequency, and the expected frequency $ind(X)$ of the independence model. The ratio $$ \frac{fr(X)}{ind(X)}$$ is called the lift of $X$. Suppose we consider patterns that have a high lift to be interesting.
Now, suppose our dataset contains $N$ transactions and the transactions are independent.  


*

*According to the independence model, the probability of generating a transaction that contains a pattern $X$ equals $fr(X)$;

*The probability $$ P(X\hspace{1mm}occurs\hspace{1mm}in\hspace{1mm}M\hspace{1mm}transactions) = {N\choose M} q^M (1-q)^{N-M},$$ where $q=ind(X)$.


Let $Z_X$ be the binomial random variable that counts the support of $X$ under the independence model.
Now, one can perform a one-sided statistical test by calculating $P(Z_X \geq Nfr(X))$. 
Because $M$ is large, one can approximate the binomial by the normal distribution $$\mathcal{N}(\mu=Nq, \sigma^2=Nq(1-q)),$$ and ultimately obtain a good approximation of the p-value of the null hypothesis of the one-sided test by calculating the probability mass of the tail of a Gaussian distribution provided that the number of transactions is very large. The most interesting patterns would be the ones that lead to the lowest p-values.
Other relevant approaches are dealt with in Chapter $5$ of Frequent Pattern Mining.
