4
$\begingroup$

I'm trying to plan a study based on previous data and I need to know the sample size required for a given effect size. Previous data looks like this:

Rating 1 2 3
control 0 20 11
treatment 6 14 12

where these are counts of the number of samples given a particular ranking at some time point in the previous study. The expected distribution of counts looks like:

Rating 1 2 3
control 2.95238095 16.73015873 11.3174603
treatment 2.95238095 16.73015873 11.3174603

and performing chi-square analysis for 2 degrees of freedom gives me chi-square value of 7.09 and a p-value of 0.029. Therefore, we conclude that the two distributions are distinct and that there is a statistically significant effect of the treatment.

I feed the chi-square value along with the table into the following function:

def getCramersV(chiStat,table):
    totalSamples = table.sum().sum()
    v = np.sqrt(chiStat / (totalSamples * (np.min(table.shape) - 1)))
    return v

and then use this value of Cramer's V (0.34) as the effect size, alpha = 0.05 and power = 0.8 in the following:

def getStatPower(effect,power,alpha):
    # perform power analysis
    analysis = TTestIndPower()
    result = analysis.solve_power(effect, power=power, nobs1=None, ratio=1.0, alpha=alpha)
    print('Sample Size needed for these parameters using TTestIndPower: %.3f' % result)

    from statsmodels.stats.power import NormalIndPower
    analysis = NormalIndPower()
    result = analysis.solve_power(effect, power=power, nobs1=None, ratio=1.0, alpha=alpha)
    print('Sample Size needed for these parameters using NormalIndPower: %.3f' % result)


    from statsmodels.stats.power import GofChisquarePower
    analysis = GofChisquarePower()
    result = analysis.solve_power(effect, power=power, nobs=None,  alpha=alpha)*2 ##times 2 because it's a one way test without it
    print('Sample Size needed for these parameters using GofChisquarePower: %.3f' % result)

The output of this is:
-Sample Size needed for these parameters using TTestIndPower: 140.488
-Sample Size needed for these parameters using NormalIndPower: 139.521
-Sample Size needed for these parameters using GofChisquarePower: 139.521

Am I misunderstanding something here either conceptually or in the code itself? If a sample size of 60ish (30 control, 30 treated) can show a statistically significant result (p-value = 0.03) and the effect size can be medium to large (Cramer's V of 0.34), how can it be suggesting that I need more than twice the sample size to observe this result 8 out of ten times? I would have thought that a sample size equal to or probably smaller should be suggested.

Can anyone clarify? Thanks in advance for any help.

$\endgroup$
0

2 Answers 2

5
$\begingroup$

As you don't provide a lot of details about the goal of your study, from the outside it looks a bit like your null hypothesis may be ill-defined:

  • why using a chi-squared test, when the variable Rating probably has an order? Don't you want to know if the treatment tend to increase or decrease the rating? The approach described by @Björn in his answer (ordinal regression) may make more sense in your case.
  • Otherwise, why do you want to detect an effect size of 0.34, and not an effect size of let's say 0.1?

Now, to answer your question about why you ended up with a required sample size bigger than what you expected, there are two things:

  1. Your code is not appropriate for sample size calculation in a context of chi-square test of homogeneity/independence.
  2. More importantly, the previous study is underpowered to detect an effect of this size, so the effect size is probably inflated. In other words the real effect size is probably smaller than 0.34. Hence, even if your code had been correct, the required sample size is anyway bigger than what was used in this previous study.

Indeed, a sample size of 63 is appropriate to detect an effect size of at least 0.39 (not 0.34) in the context of a chi-squared test:

import statsmodels.stats.power as smp
import numpy as np
my_table = np.array([[0,20,11],[6,14,12]])
smp.GofChisquarePower().solve_power(
    nobs=my_table.sum(), 
    n_bins=(2-1)*(3-1) + 1, 
    alpha=0.05,
    power=0.8)
> 0.39106456909373133

To detect an effect of 0.34, you'd need a sample size of 84 observations:

smp.GofChisquarePower().solve_power(
    0.34,                                     
    n_bins=(2-1)*(3-1) + 1, 
    alpha=0.05,
    power=0.8)
> 83.34505941708639

But the question is still: what do you find particularly interesting in an effect size of 0.34? How is it relevant to your study? Aren't there smaller effect sizes that would be interesting? For example, would you consider the following table as interesting?

Rating 1 2 3 Total
control 0% 65% 35% 100%
treatment 8% 62% 30% 100%

If so, then you're interested in an effect size of 0.2, and should adjust your sample size calculations to reflect that. But the table above is just an example: you have to think about a "minimally interesting table", and calculate its effect size. Then, run your sample size calculations based on this effect size. You'll have to think about a "minimally interesting table" no matter the test you choose to run ultimately, chi-squared or something else.


As a side note, you'll notice that in the code above, n_bins equal the degrees of freedom +1. This is a tweak necessary with the GofChisquarePower function in statsmodels in the context of a chi-squared test of homogeneity/independence. You can test the result against the pwr R library:

library(pwr)
pwr.chisq.test(w = 0.34,                    
               df = 2, 
               sig.level = 0.05, 
               power = 0.8)

Chi squared power calculation 

              w = 0.34
              N = 83.34506
             df = 2
      sig.level = 0.05
          power = 0.8
$\endgroup$
3
  • $\begingroup$ Thanks very much for your excellent response. I didn't realize that the nbins parameter needed to be set in order to use that library appropriately. The reason that I'm using the effect size of 0.34 is that this is what was calculated in the previous study, and the people I'm working with were hoping that they could decrease the sample size needed for tests of this treatment strength in the future. $\endgroup$
    – Wilhelm
    Commented Mar 7, 2023 at 18:01
  • $\begingroup$ It appears that treating these as purely categorical and using a chi-square isn't appropriate, but I still need to show whether the treatment has significant effects and what they should expect in future trials. If you have any advice on this, I would very much appreciate it. $\endgroup$
    – Wilhelm
    Commented Mar 7, 2023 at 18:02
  • $\begingroup$ @Wilhelm If you stick to a chi-square test, I doubt there's a lot of things you can do apart from increasing the sample size. If you want to explore if an ordinal regression could be a good alternative, look if there's another question treating this issue, otherwise ask a new specific question about it to get detailed answers. All the info you give here in comments, you should mention it in your original and future questions. Good luck! $\endgroup$
    – J-J-J
    Commented Mar 7, 2023 at 19:18
4
$\begingroup$

The boundary for what is "just statistically significant" (i.e. the p-value is just below some "significance threshold" such as 0.05) is, if everything you observe is the true state of nature, around the point where you would have 50% power with that sample size. This is rather obvious, when you think about it: If what you observed is exactly the truth, then by chance you have a 50% probability of a larger effect and a 50% probability of a smaller effect (unless weird boundary effects come into play).

How much larger of an effect size do you need for X% power vs. 50% power? Well, see the plot below (note the y-axis is a log-axis; code provided at the end). As you observe when you go from a sample size with a result that is "just significant" to one, where you assume everything you observed as true, to 80% power, you need slightly over twice the sample size.

enter image description here

However, there's another big point: What you observe is not the true data generating parameters. Especially, if you only proceed to the next experiment in case the first experiment is "significant", you tend to overestimate effect sizes (and this is worse the more underpowered the first experiment was). A Bayesian analysis with as realistic as possible prior distributions for the first experiment (including appropriate skepticism about whether certain effects exist or are of large size) might help in this respect. I'd be tempted to fit a Bayesian ordinal regression model e.g. with brms in R with sensible priors (see e.g. Bürkner, P.C. and Vuorre, M., 2019. Ordinal regression models in psychology: A tutorial. Advances in Methods and Practices in Psychological Science, 2(1), pp.77-101. preprint: https://psyarxiv.com/x8swp/download?format=pdf).

library(tidyverse)

tibble(`Target power` = seq(0.5, 0.99, 0.01)) %>%
  rowwise() %>%
  mutate( `Sample size ratio` = power.t.test(delta=0.01, sd=1, power=`Target power`)$n / power.t.test(delta=0.01, sd=1, power=0.5)$n) %>%
  ungroup() %>%
  ggplot(aes(x=`Target power`, y=`Sample size ratio`)) +
  theme_bw(base_size=18) +
  geom_hline(yintercept=1, linetype=2, col="darkred", size=2) +
  geom_vline(xintercept=c(0.8, 0.9), linetype=2, col=rgb(1,0.2,0.2, 0.4)) +
  geom_line(size=2, col="royalblue") +
  geom_point(size=5, col="darkblue", data=.%>%filter(`Target power` %in% c(0.5, 0.8,0.9))) +
  scale_y_log10(breaks=c(1, 1.25, 1.5, 1.75, 2, 2.5, 3, 3.5, 4, 5))
$\endgroup$
4
  • $\begingroup$ Thanks for your very helpful comments. To address/clarify in hopes of getting more information, let me say the following: * I'm having a hard time following your first paragraph: If there is a treatment that shows a large effect (a Cramer's V of 0.34 is pretty large, if I'm not mistaken) and the sample size implies that only 3% of the time would we expect to see a result like this or more extreme by random chance, I would expect more than 50% of identical trials to correctly reject the null hypothesis. Can you please clarify? $\endgroup$
    – Wilhelm
    Commented Mar 7, 2023 at 17:34
  • $\begingroup$ Also, just to give a better insight into the purpose of this, we're trying to show that a given treatment delays the number of 2 rankings relative to 1 and delays 3 relative to 2. They are ordered in that sense (1 is better than 2 is better than 3), so I can see that in treating them as pure categoricals I'm leaving information on the table. Can you please clarify how ordinal regression would allow me to show that the treatment in fact shows statistical difference between the treatment and control and how I would go about calculating effect size and the requisite sample size? $\endgroup$
    – Wilhelm
    Commented Mar 7, 2023 at 17:37
  • 1
    $\begingroup$ Related: Brunner-Munzel test for ordinal samples statsmodels.org/dev/examples/notebooks/generated/… $\endgroup$
    – Josef
    Commented Mar 7, 2023 at 19:46
  • 1
    $\begingroup$ @Wilhelm If you have a p-value just below the significance threshold (and 0.03 is pretty close to 0.05), then yes, you expect a low probability to get a significant result, if you repeated the identical experiment again. ~50% if this is the true effect, but in practice <50% because you've probably overestimated the effect. One decent explanation of this is provided in this paper. Ordinal regression uses that the categories are ordered and you can assume various things about what the effect looks like, e.g. odds ratio to move to a higher category. $\endgroup$
    – Björn
    Commented Mar 7, 2023 at 23:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.