I'm aware that this one is far from yes or no question, but I'd like to know which techniques do you prefer in categorical data analysis - i.e. cross tabulation with two categorical variables.

I've come up with:

  • χ2 test - well, this is quite self-explanatory
    • Fisher's exact test - when n < 40,
    • Yates' continuity correction - when n > 40,
  • Cramer's V - measure of association for tables which have more than 2 x 2 cells,
  • Φ coefficient - measure of association for 2 x 2 tables,
  • contingency coefficient (C) - measure of association for n x n tables,
  • odds ratio - independence of two categorical variables,
  • McNemar marginal homogeniety test,

And my question here is: Which statistical techniques for cross-tabulated data (two categorical variables) do you consider relevant (and why)?

  • 3
    $\begingroup$ This question doesn't make a lot of sense - where is your statistical/scientific question? $\endgroup$
    – hadley
    Jul 21, 2010 at 12:35

4 Answers 4


I think you need to rework this question. It all depends on the problem/data which has generated the cross-tab.

  • 2
    $\begingroup$ Yes, rather than starting with the data, start with the question and the data generating process. $\endgroup$
    – hadley
    Jul 21, 2010 at 12:35

Arguably, the question is not very precise. Rather than enumerating all measures of association for $2\times 2$ tables, I shall concentrate on the way such measures may be constructed and how to select the one that is most appropriate with respect to hypothesis or constraints relevant to a cross-classification.

The very first questions to ask are: what does the table reflect (concordance, agreement, association between two attributes, etc.), do you seek an overall measure of association or do you think one of the two variables plays a specific role (which would justify the search for an "oriented" association), do you consider either or both of the margins fixed (row and/or columns totals)? All of this impact on the method to choose and the way to interpret the results.

The $2\times 2$ case

Two-by-two tables are often treated separately from $I\times J$ tables because we often consider that variables play a symmetric role in this particular case. Obviously, this is not always the case: cross-classification of exposure and disease, as commonly found in epidemiological studies, is an example where both variables play a distinct role, which may lend to more than a simple interpretation in terms of association. Another one is $2\times 2$ tables constructed for studying the screening properties of a given diagnostic instrument. Although the odds-ratio (compared to, e.g. the relative risk) keeps its nice properties, we may be interested in predictive/negative positive values or specificity/sensibility, which means working with other quantities of interest. Hence, the need to specifiy whether the problem at hand implies two variables that are purely acting in a symmetrical way, or not, because it influences the way we interpret the results or derive a useful measure of association, agreement, or discrimination.

For the sake of clarity, I will consider that data (counts) are arranged in the following way:

alt text

Basically, measures of association for $2\times 2$ tables can be grouped in two classes: those relying on (a) (a function of) the cross-product ratio and those based on (b) the product-moment (Pearson) correlation, or a function thereof.

The cross-product ratio, mostly known as the odds-ratio, is simply $\alpha=p_{11}p_{22}/p_{12}p_{21}$. It is invariant under rows and columns interchange, and transformations of margins that preserves $\sum_{i,j}p_{ij}=1$. In epidemiology, we usually think of it as a measure of association where rows (or columns) are fixed: $p_{11}/p_{12}$ is then the odds of being in the first column (e.g., diseased) conditional on being in the first row (e.g., exposed), and likewise $p_{21}/p_{22}$ is the odds for the second row, or in other words

$$ \alpha=\frac{p_{11}/p_{12}}{p_{21}/p_{22}}. $$

Yule's $Q=(\alpha-1)/(\alpha+1)$ fall into the former case, (a). Yule also proposed a measure of "colligation", $Y$, as $(\sqrt{\alpha}-1)/(\sqrt{\alpha}+1)$. Yule's $Q$ can be interpreted as the difference between conditional probabilities of like and unlike "orders" for two individuals chosen at random; it is identical to Goodman and Kruskal's $\gamma$ measure of association for $I\times J$ tables.

For (b), we can derive a correlation coefficient for a $2\times 2$ table by thinking of the table as a combination of each of two variables scores (taking value 0 and 1, for the first and second row/column, resp.). Then, the coefficient $\rho$ is defined as the covariance divided by the square root of the product of the variances:

$$ \rho=\frac{p_{22}-p_{2\cdot}p_{\cdot 2}}{\sqrt{p_{1\cdot}p_{2\cdot}p_{\cdot 1}p_{\cdot 2}}}, $$

which is equivalent to putting $p_{11}p_{22}-p_{21}p_{12}$ in the numerator. Plugging in the observed counts, Pearson's $r$ is the MLE of $\rho$ under a multinomial sampling model. It is invariant under rows and columns interchange, and positive linear transformation.

It can be shown (Yule, 1912) that $\rho$ is identical to Yule's $Y$ if we standardize our table such that row and column margins sum to 1/2, i.e. $p_{11}^*=p_{22}^*=0.5\left(\sqrt{\alpha}/(\sqrt{\alpha}+1)\right)$ and $p_{12}^*=p_{21}^*=0.5\left(1/(\sqrt{\alpha}+1)\right)$. By doing this, we remove the information coming from the margins, such that $Y=2(p_{11}^*-p_{12}^*)$.

Correlation-based measures are connected to the usual Pearson's chi-square statistic, since

$$ \Phi^2=\sum_{i=1}^2\sum_{j=1}^2\frac{(p_{ij}-p_{i\cdot}p_{\cdot j})^2}{p_{i\cdot}p_{\cdot j}}, $$

that is,

$$ \Phi^2=\frac{(p_{11}p_{22}-p_{21}p_{12})^2}{p_{1\cdot}p_{2\cdot}p_{\cdot 1}p_{\cdot 2}}=\rho^2. $$

In a $2\times 2$ table, we thus have $r^2=\chi^2/N$. Pearson also proposed to use $\sqrt{\rho^2/(1+\rho^2)}$ as a measure of association, and he coined it the coefficient of mean square contingency.

As to how to choose the correct measure (a vs. b), it clearly depends on whether we want to be sensitive to marginal totals (in this case, $\rho$ cannot take its full range of possible values in $[-1;1]$), and whether we consider that we observe a full association even if one of the four cells is zero (in this case, $\rho$ cannot take the value $+1$ or $-1$ if only one of the cells is zero, which is not the case of Yule's $Q$). Of note, correlation-based measures are better if they are used in a correlation matrix (e.g., for factor analysis), because we cannot guarantee that a matrix composed of Yule's $Q$ coefficient will be positive definite.

The $I\times J$ case

Like for the $2\times 2$ case, we can derive measures of association based on different quantities. Measures based on chi-square include

  • Pearson's $P$ coefficient based on $\Phi^2$ (see above), $\sqrt{\Phi^2/(\Phi^2+1)}$ (to overcome the fact that $\Phi^2$ no longer lies in $[0;1]$ when $I$ or $J>2$);
  • Tschuprow's $T=\left(\Phi^2/\sqrt{(I-1)(J-1)}\right)^{1/2}$, which behaves better than $P$ in square tables (in that it can reach a maximum value of 1, for full or complete association);
  • Cramer's $V$ is another derivation, and $V=\left(\Phi^2/\text{min}(I-1,J-1)\right)^{1/2}$ (we have $V\geq T$ for all $I,J>2$).

These measures are all measures of association where none of the variables plays a specific role. In case a $\chi^2$ test is significant, it is more interesting to look at how the expected counts depart from the observed counts (i.e. look at the Pearson residuals) in all $(i,j)$ cells, or to use something like a mosaic plot.

Goodman and Kruskall (1954) also proposed a predictive measure of association between rows and columns, or more specifically a measure of proportional reduction in error in predicting one column category when the row category is known as opposed to the case when the latter one is unknown. This is called $\lambda_{C|R}$ and its MLE is

$$ \hat\lambda_{C|R}=\frac{\sum_{i=1}^Ix_{im}-x_{\cdot m}}{N-x_{\cdot m}} $$

where $x_{im}$ and $x_{\cdot m}$ stand for the maximum for the $i$th row and the column totals. This measure is interesting because it has a nicer interpretation than $\chi^2$-based measure, but it also has some drawbacks: when there is statistical independence, $\lambda_{C|R}$ is not necessarily zero, for instance.

A measure of the proportion of explained variance (derived from Gini's total variation) may be derived from the total sum of squares (SS) in an $I\times J$ table

$$ \text{TSS}=\frac{N}{2}-\frac{1}{2N}\sum_{i=1}^Ix_{i\cdot}^2, $$

which can be partitioned as a within- and between-group SS. Of interest here is the variance explained by considering the different categories (BSS) divided by the total variance, TSS. Like in the ANOVA framework, we have BSS=TSS-WSS, where

$$ \text{WSS}=\frac{N}{2}-\frac{1}{2}\sum_{j=1}^J\frac{1}{x_{\cdot j}}\sum_{i=1}^Ix_{ij}^2, $$

so that we can derive BSS/TSS as

$$ \hat\tau_{R|C}=\frac{\sum_j\frac{1}{x_{\cdot j}}\sum_i x_{ij}^2-\frac{1}{N}\sum_ix_{i\cdot}^2}{N-\frac{1}{N}\sum_ix_{i\cdot}^2}. $$

This measure can be interpreted as "the relative decrease in the proportion of incorrect predictions when we go from predicting the row category based only on the row marginal probabilities to predicting the row category based on the conditional proportions $p_{ij}/p_{\cdot j}$" (Bishop et al., 2007, p. 391).

Finally, measures based on the cross-product ratios are also available, as well as measures of agreement for ordinal variables, but I realize now that I need to stop (and thank the reader who reached the end of this overview).

A thorough overview of measures of association may be found in Bishop et al. (2007), from which I grabbed most of the above discussion, and of course Agresti (2002), about which Laura Thompson made a complete R adaptation in his textbook R (and S-PLUS) Manual to Accompany Agresti's Categorical Data Analysis.


  1. Agresti, A. (2002). Categorical Data Analysis. Wiley. Companion website
  2. Bishop, Y.M., Fienberg, S.E., and Holland, P.W. (2007). Discrete Multivariate Analysis. Springer.
  3. Goodman, L.A. and Kruskall, W.H. (1954). Measures of association for cross-classification. JASA, 49, 732-764.
  4. Yule, G.U. (1912). On the methods of measuring association between two attributes. Journal of the Royal Society, 75, 579-642.

I would use Fisher's Exact Test, even for large N. I wouldn't know why not. Any performance argument predates today's fast computers.

  • 2
    $\begingroup$ I agree that computing resources allow to easily perform permutation and exact tests, or long runs of Monte Carlo simulations, but here is a paper (from Statistics in Medicine) where the authors argued against the systematic use of Fisher's exact test and rather recommended choosing the test according to the question that is posed: bit.ly/9qflht. $\endgroup$
    – chl
    Sep 16, 2010 at 12:20
  • $\begingroup$ @chl: I don't have (free) access to the article. I'm very interested in the arguments. What can be more exact than Exact? $\endgroup$ Sep 19, 2010 at 20:11
  • 2
    $\begingroup$ Here is the electronic copy in case you want to read it (it's very instructive and includes a lot of references): j.mp/b2Wbpn. $\endgroup$
    – chl
    Sep 19, 2010 at 20:20
  • 1
    $\begingroup$ Why not? Because Fisher's Exact Test it assumes that data generating process has both sets of marginal totals fixed but actually occurring processes seldom do. As Hadley points out, we need to know about that generating process. $\endgroup$ Nov 3, 2010 at 12:59
  • $\begingroup$ Now that I think of it: agreed! $\endgroup$ Nov 8, 2010 at 14:38

I must agree.. there is no single best analysis! not just in cross tabulations or analysis of categorical data but in any data analysis... and thank god for that! if there was just a single best way to address these analyses well many of us would not have a job to start with... not to mention the loss of the thrill of the hunt!

the joy of analysis is the unknown and the search for answers and evidence and how one question leads to another... that is what i love about statistics!

So back to the categorical data analysis... it really depends on what your doing. Are you looking to find how different variables affect each other as in drug tests for example we may look at treatment vs placebo crossed with disease and no disease... the question here is does treatment reduce disease.... chi square usually does well here (given a good sample size). Another context ihad today was looking at missing value trends... i was looking to find if missing values in one categorical variable relate to another... in some cases i knew the result should be missing and yet there were observations that had values... a completely different context to the drug test!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.