# Finding anomaly using statistical distributions [closed]

I am trying to find if a particular genre of books is more popular with people of a particular region of the world. I have data for the readership of a set of books of different genre corresponding to different countries of the world. It is essentially sales of the book.

Suppose there are N genres. I am interested in a particular genre and wish to determine if this genre's readership is different from the 'mean' readership. How could determine the mean distribution of readership across all genres? Then the distribution of the readership of a particular genre. And then compare them mathematically and determine if there is a difference.

I wish to make a conclusion in the lines of This genre is more popular in these set of countries with a confidence value.

Another related question is, what is the method for determining if the data points I have are enough to do this analysis.

If there is some other line of thought to arrive at the desired conclusion (or the lack of it), I am keen on knowing about it.

• If you have the data in a contingency table with genres x countries, maybe start with correspondence analysis. Aug 9 '20 at 19:39
• @kjetilbhalvorsen A cursory look at correspondence analysis on Wikipedia says it is essentially PCA for categorical values. I actually have numerical values for readership~sales. But thanks for pointing it out - I did not know about this either. And I could look at PCA itself for this problem. Aug 9 '20 at 20:06
• While CA has been compared as PCA for categorical values, it has a rather different flavour! Aug 9 '20 at 20:34

One approach would be to use a chi-square test.

To do this, you would estimate the overall distribution of sales in various genres from all of the data. For example, say your data revealed this distribution:

Non-fiction science/industry: 20%
Cookbooks: 35%
Speculative fiction (sci-fi, fantasy, etc.): 25%
Crime fiction: 10%
Other fiction: 10%


You then get data of 15,000 sales from Lostralia with a different distribution:

Non-fiction science/industry: 25%
Cookbooks: 25%
Speculative fiction (sci-fi, fantasy, etc.): 35%
Crime fiction: 10%
Other fiction: 5%


How to assess if this is significantly different from the overall distribution? Compute the $$\chi^2$$ test statistic: $$X^2 = N \sum_{i=1}^5 \frac{(\textrm{observed freq.} - \textrm{expected freq.})^2}{\textrm{expected freq.}}.$$ The $$X^2$$ test statistics is a $$\chi^2$$ variable with 4 degrees of freedom (because once you know the first four frequencies, you know the fifth).

In this example, our $$X^2$$ test statistic is \begin{align} X^2 &= 15000\left(\frac{(0.25-0.20)^2}{0.20}+\frac{(0.25-0.35)^2}{0.35}+\frac{(0.35-0.25)^2}{0.25}+\frac{(0.10-0.10)^2}{0.10}+\frac{(0.05-0.10)^2}{0.10}\right) \\ &= 15000 \cdot 0.10607 = 1591.07 \end{align}

Perhaps you only care about cookbooks. In that case, collapse all the other genres into an "everything else" category and compute $$X^2$$ (now 1 degree of freedom) for that: \begin{align} X^2 &= 15000\left(\frac{(0.25-0.35)^2}{0.35}+\frac{(0.75-0.65)^2}{0.65}\right) \\ &= 15000 \cdot 0.04396 = 659.34 \end{align}

You don't need to run $$R$$ and type 1-pchisq(659.34, df=1) to guess that this is highly significant. Unfortunately, with any reasonably large amount of sales data you are likely to conclude a difference from the overall distribution of sales.

However, since you plan on doing multiple comparisons (which countries are different from the overall distribution), you could do a Bonferroni correction on what is an acceptable $$p$$-value by dividing it by the number of countries $$c$$ you check. So if $$c=5$$ you would conclude 5% significance if $$p\leq 0.01$$.

One weakness of this approach is also taking the overall distribution of sales as though it has no uncertainty. I suppose you could use the uncertainty of that overall distribution to simulate possible distributions, compute $$X^2$$ test statistics, and then find a way to combine those test statistics to get a better test statistic.

• Could you kindly elaborate a bit more on the last paragraph of your answer? Aug 10 '20 at 6:05