# Problem with $\chi^2$ test

I am very new to statistic analysis and I am having some problems with my data. Basically, my categorical variables are the following: grammatical structures (four of them) and languages (five of them). The total number of these structures is 1218.

I want to answer the research question if the observed preference shown by some languages for certain grammatical structures is merely due to random chance or there is a relationship between them. I applied the chi-square test and I obtained the following p-value: 1.5168E-140 (chi-square value = 692.813, due to a huge difference between observed and expected frequencies, and a df = 12).

I am really puzzled by this result. My guess is that the data are too dispersed and the chi-square test is inconsistent here. So does that actually mean that the sample is biased or that it does not make much sense to answer that particular research question in the beginning? Or maybe should I use some other test? Thanks in advance.

• There's simply not enough information here about what you did to hazard a good answer. – Glen_b Mar 22 '13 at 22:39
• Can you please sketch the contingency table for your data? – Zen Mar 23 '13 at 1:28
• This means your result is highly significant. Is that a problem? – Jeremy Miles Mar 23 '13 at 14:24

If the $\chi^2$ statistics are biased will depend on various issues that it is currently impossible from your question to infer.
In general, you want to have random sampling from a population and also a sufficient amount of frequencies for each cell (due to rounding errors in discrete values). The latter condition can be subjective regarding "what is sufficient", but simulations revealed that $>5$ for each cell is a good rule of thumb. The random sampling ensures that independent observation will have a mean that is approximately normally distributed which is a prerequisite for calculation of the $\chi^2$ test statistics. All this provides a good set up for obtaining unbiased $\chi^2$ statistics.
Given that you have taken care of all this, $\chi^2$ ought to be be relatively unbiased. And do not worry, the dimension of the p statistics (so close to zero) you have shown is quite frequent in $\chi^2$ tests with large data sets - even when you have very small differences between expected and actual value. This is because the test becomes very accurate in large samples, and you can confidently claim, for example, 10 is (statistically) smaller than 10.01. A large sample quickly shoots up the $\chi^2$ statistics even for small differences in expected and actual values, which is the nature of the test and not a bias by itself.
So there is a good chance that your test statistics are actually right, but as I said, you need to make sure that your categories are aptly chosen, that is they are mutually exclusive, exhaustive and best if evenly frequent. Furthermore, for your observations you must ensure a random sampling from the population. (Of course, if you don't have a sample but you take the whole population into account, you won't need a $\chi^2$ test in the first place.)