I have two frequency tables, one representing observed data and one representing modeled data.

I am looking for a Goodness of Fit measure, checking whether the model data fits the observed data.

Problem is, my counts are rather small (most are smaller than 5), and so Pearson Goodness of Fit fails to operate on these data sets (I am using R).

R reports the following:

X-squared = Inf, df = 534, p-value < 2.2e-16

Should I use another Goodness of Fit measure? Any suggestions?

EDIT (providing some more info):

The model contains frequencies of "topics" that are associated with "documents" read by users in the last month.

The observed data contains "topics" that are associated with "documents" read by users in the last day.

(If "documents" are Music CDs, "topics" are musical Genres, for example)

I am trying to understand if the model (data over time) fits the observed data (data of last day). If the model fits the observed data, it should mean that users stay around the same "topics" (keep listening to the same musical genres).

  • $\begingroup$ What variables are these? df = 534 indicates a 2x268 table. What variable has 268 categories? Can these be combined somehow? There are exact tests, but with such a large table they may take a very long time to run. $\endgroup$ – Peter Flom Sep 6 '13 at 13:44
  • $\begingroup$ The data represents topical labeling of documents. $\endgroup$ – Ido Tamir Sep 6 '13 at 14:09
  • $\begingroup$ Can you tell us more about what you are trying to do? this post on my blog may help $\endgroup$ – Peter Flom Sep 6 '13 at 14:13
  • $\begingroup$ Provided some more info - hope it helps. $\endgroup$ – Ido Tamir Sep 6 '13 at 15:05
  • $\begingroup$ Your expecteds simply seem to be observeds from a different sample. Since your other, 'sample' sample has observeds in different cells to that earlier sample, you will automatically get a chi-square of infinity. This seems to be simply caused by your error of treating something that isn't a population as a population. If it really is your population, the chi-square of infinity is exactly right, because you have observed an infinitely unlikely event (an event of probability zero occurred). My suggestion is instead to treat both samples as samples and test them for equality of distribution. $\endgroup$ – Glen_b Sep 7 '13 at 0:31

Small expected counts may or may not be the main issue here. The chi-square statistic is reported as infinite. At a guess, one or more expected counts are zero, at least to machine precision. It is unlikely that chi-square is appropriate without some recasting of the problem.

Chi-square testing of goodness of fit classically arises with a relatively number of predefined classes or with classes produced by binning of data. A problem with 534 df is not out of the question, but on no precise information, as here, I would guess that your number of classes is just too large, somehow.

(UPDATE) It sounds as you are using observed past frequencies as if they were expected frequencies, but if any past frequency was zero, chi-square is indeterminate. The proper test is to use past and present frequencies as two observed frequency vectors, but even then the number of categories is likely to be too large for that to be practical. I would favour a much more exploratory approach in which you plot frequencies and look for big discrepancies. A square root scale for frequency would temper variation while allowing zeros to be plotted.

In addition, the total observed and expected frequencies must match.

  • $\begingroup$ I might be applying the wrong tool for the issue at hand. Any other method (other than Chi-square) I should be looking into? $\endgroup$ – Ido Tamir Sep 6 '13 at 14:10
  • 1
    $\begingroup$ Sorry, but trying to provide advice would be foolish without much more information on what you are doing (as @Peter Flom has also requested). $\endgroup$ – Nick Cox Sep 6 '13 at 14:20
  • $\begingroup$ I added some more info - hope its more clear now. $\endgroup$ – Ido Tamir Sep 6 '13 at 15:05
  • $\begingroup$ Correct, zero past frequency makes chi-square indeterminate. I'll mark this answer as correct, as it provided some insight into my issue. Thanks. $\endgroup$ – Ido Tamir Sep 6 '13 at 18:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.