chi-squared goodness-of-fit: effect size and power

Question

How can effect size and power of the chi-square goodness-of-fit test be calculated such that:

1. Effect size is linear and standardized (given effect size $E_{x^2}$, $0 \le E_{x^2} \le 1$) so that tables are not required to determine if the effect size is small, medium, or large. This would be used to validate a significant effect size when the p-value of the test is less than a given $\alpha$.

2. the power can be used to calculate the required sample size given a desired false negative rate $\beta$.

The only restriction is that my test variable may have $2^{32}$ different levels. See chi-squared with too many degrees of freedom for more information.

What follows is the research I've done towards answering my question. Keep in mind it may be completely unfounded, but since it covers my knowledge on the topic I would appreciate if answers would take it into consideration.

I've found the following formula of power:

$$1 - \beta = F_{df,\lambda}(x_{crit})$$ where

• $\alpha$ is the false positive rate
• $\beta$ is the false negative rate
• $F_{df,\lambda}$ is the CDF of the non-central chi-square distribution $x^2_{df}$

• $x_{crit}$ is the critical value for the given value of $\alpha$. I assume this means:

  $1 - F_{df}(x_{crit}) = \alpha$

  where

  • $F_{df}$ is the CDF of the chi-square distribution $x^2_{df}$

• $\lambda$ is the noncentrality parameter such that

  $\lambda = \phi^2n$

  where

  • $\phi$ is the effect size
  • $n$ is the sample size

I just don't know which effect size $\phi$ references:

1. The Phi Coefficient, $\sqrt{x^2/N}$

   $\phi$ is only applicable to binomial variables, ie 2x2 contingency tables. Therefore, it probably isn't suited to goodness-of-fit tests.

2. Cramér's $\phi$, $\sqrt{\frac{x^2}{N(k-1)}}$

   $\phi_c$ "may be used with variables having more than two levels", and is therefore suited to goodness-of-fit (1xk tables) tests. Unfortunately $\phi_c$ requires tables to determine the significance of the effect size (see this question), and may be meaningless for large $k$. Apparently it "functions as a measure of tendency towards a single outcome", ie between the variables Expected and Observed. Unfortunately I can't find any references that can confirm $\phi_c$ is meaningful for the goodness-of-fit test (see this question). In fact, the standard definition of Cramér's $\phi$:

   $\phi = \sqrt{\frac{x^2}{n*min(c-1,r-1)}}$

   leads to undefined behaviour:

   $\phi = \sqrt{\frac{x^2}{0}}$

3. Cohen's w, $\sqrt{\sum_{i=1}^N \frac{(p_{0i} - p_{1i})^2}{p_{0i}}}$

   This looks like $w = \sqrt{x^2}$, which implies it isn't standardized such that $0 \le w \le 1$. Other than that, I don't know much about it.

4. $E_{x^2}$ ^[1], $\frac{x^2q}{n(1-q)}$

   This simply standardizes $x^2$ from the goodness-of-fit test against its maximum, ie:

   $x^2 = n \sum_{i=1}^k \frac{p_i^2}{q_i} - 1$

   where

   • $k$ is the number of categories
   • $n$ is the sample size

   • $p_i$ is the observed category proportion, I interpret this as:

     $p_i = O_i / n$

     where

     • $O_i$ is the observed category frequency from the standard definition of $x^2$

   • $q_i$ is the expected category proportion, I interpret this as:

     $q_i = E_i / n$

     where

     • $E_i$ is the expected category frequency from the standard definition of $x^2$

   
   

   $x^2_{max} = n \left( \frac{1-q}{q} \right)$

   $E_{x^2} = \frac{x^2}{x^2_{max}} = \frac{x^2q}{n(1-q)}$

   Unfortunately, I can't reconcile the paper's definition of $x^2$ against the standard definition:

   $x^2 = \sum_{i=1}^n \frac{(O_i - E_i)^2}{E_i} = N \sum_{i=1}^n q_i\frac{(p_i - q_i)^2}{q_i} = N \sum_{i=1}^n \frac{p_i^2}{q_i} - 2p_i + q_i$

In any case, I would assume that any definition of power would be tied to a particular measure of effect size.

[1] J. E. Johnston, Kenneth J Berry, Paul W Mielke. (2006) Measures of effect size for chi-squared and likelihood-ratio goodness-of-fit tests

Answer 1

$\lambda =\omega^2N$, see Cohen, Jacob (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.), page 549, formula 12.7.1.

Hence the effect size you mention is Cohen's omega ($\omega$, sometimes written "w").

$\omega=\sqrt{\frac{\chi2}{N}}$. The $p_{0i}$ and $p_{1i}$ in the formula you give in your question are proportions, not counts. $\omega$ generally has not an upper bound of 1 (except when there are just two cells of expected proportions of 0.5), but it's not really a problem if you use it for sample size calculations.

Similarly, you don't need tables to define an effect size of minimal interest. In fact in his book Cohen tends to advise against using them, see his introductory remarks of section 7.2.3 of his book, p. 224:

[...]

The best guide here, as always, is the development of some sense of magnitude ad hoc, for a particular problem or a particular field. Since it is a function of proportions, the investigator should generally be able to express the size of the effect he wishes to be able to detect by writing a set of alternate-hypothetical proportions [...] and, with the null-hypothetical proportions, compute w.

[...]

In other words, you have to define what a "minimally interesting table" would look like in your scenario. For example, you can ask yourself how many cells should deviate from their expected value and by how much in terms of proportions, for you to consider such a table as an interesting departure from the null hypothesis.

Once you defined this hypothetical, minimally interesting table, you calculate its effect size $\omega$. From that, you can calculate the sample size required to detect this "minimally interesting effect size".

---

As for the other effect sizes you mention, Cramér's V and Phi are meant as measures of association, in other words they are meant for contingency tables, not one-way tables. There's a variant of Cramér's V for one-way tables, but it does not really bring a lot more information than Cohen's $\omega$, and is not meant for power calculations.

I'm not familiar with the Johnston-Berry-Mielke $E$, but from the article you mention its purpose seems to be an easier interpretation than $\omega$, not power calculations.