2
$\begingroup$

I am very new to statistic analysis and I am having some problems with my data. Basically, my categorical variables are the following: grammatical structures (four of them) and languages (five of them). The total number of these structures is 1218.

I want to answer the research question if the observed preference shown by some languages for certain grammatical structures is merely due to random chance or there is a relationship between them. I applied the chi-square test and I obtained the following p-value: 1.5168E-140 (chi-square value = 692.813, due to a huge difference between observed and expected frequencies, and a df = 12).

I am really puzzled by this result. My guess is that the data are too dispersed and the chi-square test is inconsistent here. So does that actually mean that the sample is biased or that it does not make much sense to answer that particular research question in the beginning? Or maybe should I use some other test? Thanks in advance.

$\endgroup$
  • $\begingroup$ There's simply not enough information here about what you did to hazard a good answer. $\endgroup$ – Glen_b Mar 22 '13 at 22:39
  • 1
    $\begingroup$ Can you please sketch the contingency table for your data? $\endgroup$ – Zen Mar 23 '13 at 1:28
  • 1
    $\begingroup$ This means your result is highly significant. Is that a problem? $\endgroup$ – Jeremy Miles Mar 23 '13 at 14:24
1
$\begingroup$

If the $\chi^2$ statistics are biased will depend on various issues that it is currently impossible from your question to infer.

In general, you want to have random sampling from a population and also a sufficient amount of frequencies for each cell (due to rounding errors in discrete values). The latter condition can be subjective regarding "what is sufficient", but simulations revealed that $>5$ for each cell is a good rule of thumb. The random sampling ensures that independent observation will have a mean that is approximately normally distributed which is a prerequisite for calculation of the $\chi^2$ test statistics. All this provides a good set up for obtaining unbiased $\chi^2$ statistics.

Now comes the tricky part: You need to ensure that each cell is mutually exclusive of each other and that you cover ALL categories (exhaustively). It is not entirely clear from your question if that is the case for the variable grammatical structure or language. For example, Italian, German, Japanese are all languages, but do you have a categories for the rest, so to speak "all other languages" as well? Furthermore, is are the grammar categories mutually exclusive? I am no linguist but I can imagine that sometimes there is no clear cut here. Also what are your actual observations here, where does data for your study originate from?

Given that you have taken care of all this, $\chi^2$ ought to be be relatively unbiased. And do not worry, the dimension of the p statistics (so close to zero) you have shown is quite frequent in $\chi^2$ tests with large data sets - even when you have very small differences between expected and actual value. This is because the test becomes very accurate in large samples, and you can confidently claim, for example, 10 is (statistically) smaller than 10.01. A large sample quickly shoots up the $\chi^2$ statistics even for small differences in expected and actual values, which is the nature of the test and not a bias by itself.

So there is a good chance that your test statistics are actually right, but as I said, you need to make sure that your categories are aptly chosen, that is they are mutually exclusive, exhaustive and best if evenly frequent. Furthermore, for your observations you must ensure a random sampling from the population. (Of course, if you don't have a sample but you take the whole population into account, you won't need a $\chi^2$ test in the first place.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.