6
$\begingroup$

I need to compare sample distributions with theoretical ones, which is typically done with a chi-squared test. The problem is that I have distributions where one or more cells have low values, and consequently the chi-squared test reports very small p-values. For example, a typical expected and observed frequencies are [152 2 9] and [140 5 18], with a p-value of 0.0007. Based on domain knowledge, these two distributions are not significantly different.

What test could be used instead of chi-squared, which would take out the bias that occurs with the small-valued cells?

Edit: adding some background information for this problem.

I have a number of processes which produce as output certain technical parameters, recorded as time series. I have around 4000 of such process, each producing around 150 such time series (the number of time series a process has follows a power law). I would like to find which of these processes are anomalous, i.e. producing output which is significantly different from others. To do this I cluster the time series using k-means, and then based on the clusters, produce the "expected" distribution (average over all time series) and the distribution of clusters for each process.

For example, after clustering I might have 4 clusters with following sizes.

Cluster number | Cluster size
-----------------------------
1              | 100
2              | 200
3              | 300
4              | 400

The distribution of the clusters among the processes might be the following

          | Cluster 1 | Cluster 2 | Cluster 3 | Cluster 4
----------------------------------------------------------
Process 1 | 11        | 19        | 35        | 42
Process 2 | 3         | 10        | 14        | 19
Process 3 | 30        | 8         | 12        | 12              <----anomaly
....

In this case, process 1 and process 2 are sufficiently close to the expected, while process 3 has a different distribution from the average. I would like to find a good test to measure this discrepancy. (Any other suggestion for the anomaly detection is also welcome)

$\endgroup$
  • 2
    $\begingroup$ It is noteworthy that the edit almost completely changes the question. (This is no longer a comparison of a distribution against a reference: it involves multiple comparisons and no reference.) There are now good reasons to suspect correlations among the counts, too, which could be induced by the nature of the clustering and the nature of the processes themselves. Answers can be derived by ignoring these complications, but they might then lead you to erroneous conclusions. A good answer would incorporate some model (or understanding) of the relationships between processes and clusters. $\endgroup$ – whuber Apr 2 '12 at 15:11
4
$\begingroup$

There are two technical issues to deal with: (1) measuring the discrepancy between observed and expected and (2) computing the p-value.

We can retain the chi-squared measure of discrepancy (thereby finessing issue 1) and compute an exact p-value. The simple way is to simulate sampling from the expected distribution. Here is the distribution of 10,000 samples performed in R:

Histogram

The actual chi-squared statistic for these data is $549/38 \approx 14.447$. Apparently it is far out in the upper tail of this histogram: only $25$ of the $10,000$ results (0.25%) equal or exceed it. Yes, this proportion is almost four times greater than the approximation of $0.0007$ reported by the chi-squared test, but it's still tiny. We conclude that the observed distribution is significantly different from the expected distribution.

The "domain knowledge" may indeed correctly suggest the amount of difference is not material. That, however, is independent of the finding that the observed frequencies are unlikely to arise randomly from a distribution with the expected frequencies. That is all that statistical significance means.

$\endgroup$
  • $\begingroup$ Thanks for the reply. Could you also elaborate on point (1). I.e. why the chi-squared measure in the first place. Could another test statistic to measure the discrepancy be more useful? $\endgroup$ – Ash Apr 1 '12 at 20:21
  • $\begingroup$ Yes, Ash, another statistic could be more useful. That would depend why you are conducting this test, about which I and all other readers are ignorant. If you could like to elaborate on that, perhaps we can make more progress in this direction. It might also be helpful to disclose the basis of the expected frequencies: if they cannot be taken as given by theory, but are rather derived from other data or assumptions, then we have failed to incorporate that additional uncertainty in our reasoning. $\endgroup$ – whuber Apr 1 '12 at 20:32
  • $\begingroup$ Added some background info to the question $\endgroup$ – Ash Apr 1 '12 at 21:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.