4
$\begingroup$

I have done an experiment to test the performance of a system. In this experiment I collected answers from people. These answers are categorical (they were able to select on a 5 points scale). Each subject participated to 2 conditions: in one they are giving evaluations for observations under an ideal situation and in the other they are giving evaluations for observations coming from the real results of my system. My hypothesis is that the distribution of the ideal condition fits the distribution well under the real condition, so I want to demonstrate that the results of my system are perceived equally good as in the ideal condition by users.

My results for the 5 points of the scale are the following:

Observed frequencies (real condition): {16, 112, 42, 308, 100}

Expected frequencies (ideal condition): {9, 81, 53, 340, 95}

As I saw that most of the time the chi squared goodness of fit test is used with an expected frequency that comes from laws and not from empirical observations, I was wondering if:

  1. My approach of using the chi squared test could be good for my aim, or if there is another better tool;
  2. If it is okay to use empirical frequencies given by the observation of the same population under another condition.
$\endgroup$
  • 1
    $\begingroup$ The usual approach would be to do a chi-squared test of homogeneity, which involves looking at it as a $2\times k$ table - thus treating both sets of empirical observations as, well, observations. $\endgroup$ – Glen_b -Reinstate Monica Jan 29 '14 at 9:03
2
$\begingroup$

As @Glen_b pointed out, if you want a $\chi^2$ test, you want the test of homogeneity. The goodness of fit (GoF) test essentially compares your observed frequencies to any model you want it to. Stat Trek states of the GoF test, "The test is applied when you have one categorical variable from a single population."

These models don't come from laws all the time in accepted usage, and often do come from empirical observations, as when comparing one empirical distribution to a model of equal frequencies across all categories. In the latter case, one would probably just use the mean frequency of all observations to fill each expected frequency cell in one's theoretical model. Even this doesn't have to be within the researcher's control, as one could decide when to stop observing after a certain period of observation had elapsed rather than deciding in advance how many observations to record.

When comparing two empirical distributions of categorical data like you have here, I think you'll get the same numbers with either test, but you'll calculate the degrees of freedom slightly differently (to achieve the same result). In the GoF test, $df=k-1$ where $k$ is the number of levels in your variable. In the test of homogeneity, $df=(r-1)\cdot(c-1)$ where $r$ (rows) is the number of populations you have, and $c$ is just like the GoF test's $k$. In a two-population case like yours, $r-1=1$, but if you had a third population, you couldn't use the GoF formula anymore, whereas the test of homogeneity would still work fine.

If you're aiming to confirm your hypothesis, the $\chi^2$ test might not please you, because its result supports rejecting the null hypothesis, which is literally that your two distributions were sampled randomly from identically distributed populations. Based on the data you've given: $$\chi^2_{(4)}=22.867, p<.001$$ If you were aiming to truly test the null hypothesis as I've stated it, there you go! If your two distributions were sampled randomly from identically distributed populations, you have less than a one-in-1000 chance of sampling another set of two distributions that are at least as different as the two samples you have here. (BTW, even when a significance test gives $p>.05$, it doesn't confirm the null hypothesis per se; it fails to reject it.)

If you want to judge whether your distribution from the ideal condition is a good model of the real condition, you may want to define "good model" somewhat more loosely than this, and maybe ask the question, how similar are my two distributions? This is not a matter of the probability of the null hypothesis (which seems unlikely to be literally true in any case), but of estimating the strength of the relationship or effect size. In your case, you could calculate Cramér's $\phi$ quite easily: $$\sqrt{\frac{\chi^2}{N(k-1)}}=.14$$ This $k$ is the lesser of $r$ and $c$, which is $2$. A $\phi=1$ would represent exact identicalness, and $\phi=0$ means no association. @gung pointed out elsewhere that interpreting this estimate depends on the size of the contingency table ($\phi$ tends to increase with more cells; BTW, gung mentioned some other problems with $\phi$, and alternatives too), but with only ten cells, you probably haven't got too much upward biasing here, and it's a fairly weak relationship either way.

Still, this treats your data as , and if the points on your scale are ordered, your data are actually , making them somewhat more information-rich, and the aforementioned methods don't take advantage of that at all. If each of your participants was actually evaluating similar observations under two conditions each (e.g., "Rate this observation under the real condition, then rate that same observation under the ideal condition"), your data are also paired across your condition variable. In this case, you want your analysis to recognize that one observation can be evaluated similarly across both conditions without being evaluated identically (e.g., fifth rating in real condition, and fourth rating in ideal condition, as opposed to fifth and first, if that's a pair of opposite evaluations).

To test the strength of association between ordinal evaluations that are paired within individuals across two conditions, you again have some alternatives to choose from, including Spearman's $\rho$, Kendall's $\tau$, and the Goodman-Kruskal $\gamma$. If you have paired ordinal data, and need to choose which statistic to calculate, you might have a look at these questions:

As I read them, $\tau$ seems somewhat preferable to $\rho$ in many cases, and fairly similar to $\gamma$, but converting $\tau$ to the scale of Pearson's $r$ requires this formula: $$\begin{aligned}r=\sin\bigg(\tau\cdot\frac\pi2\bigg)\end{aligned}$$ Conventional guidelines for interpreting the magnitude of $r$ (which can range from $1$ to $-1$ in value):

  • $r=.1$ is a weak correlation
  • $r=.3$ is a moderate correlation
  • $r=.5$ is a strong correlation
  • $r>.71$ means your variables are $>50\%$ similar (i.e., proportion of shared variance, $r^2>.5$)

If you can calculate an effect size like $\tau$ and scale it to $r$, you may want to decide just how similar you want your distributions to be to call the fit "good", and compare that to the value of that $r^2$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.