1
$\begingroup$

I want to confirm whether my interpretation makes sense, hope you can guide.

I'm testing the effect of an email intervention on an outcome (attendance rates to an event).

Main Group Observations

  • Turnout rate was lower with intervention than without (Odds ratio was 0.7)
  • Difference is statistically significant.
  • (yes, sadly the intervention did not work out)

Subgroup of Females Observations

  • Turnout rate was lower with intervention than without, but the rate was slightly higher than for females (Odds Ratio was 0.8)
  • Difference is NOT statistically significant.

Can I argue that the female subgroup is more receptive to the intervention, given that it (a) has the higher rate, and (b) it has a nonsignificant difference?

Reason I'm asking: While it feels 'intuitive' - I'm not sure if there's something I'm missing out, especially when the subgroup has a smaller sample size < and whether this sample size can mess up interpretations.

Improve this question
$\endgroup$
1
  • $\begingroup$ Even without making statistical implications, claiming that the females were "more receptive" to the intervention seems disingenuous. A more appropriate description is that they were "less unreceptive". $\endgroup$
    – dipetkov
    Apr 5, 2022 at 0:00

1 Answer 1

Reset to default
0
$\begingroup$

If all you want to do is describe what occurred in your sample (i.e., without any inference), then your conclusions are correct as long as you eliminate all discussion of statistical significance and do not attempt to make any claims about the performance of the intervention in other samples.

If you want to perform inference (i.e., talk about what the effect of the intervention is, rather than was), you need a formal test for any comparison you wish to interpret. If you want to make a claim about the difference in the effect between males and females, you need to perform a statistical test to make such a claim. It is critical that you do not interpret the lack of significance as a lack of effect; it simply means you failed to find an effect, not that there isn't one. The two reasons you might fail to find an effect is that there isn't one or there is one but it is too small for your sample to detect.

It is also worth interpreting the confidence intervals around your estimates. Does the confidence interval indicate that there is no effect, or are substantial effect sizes contained within the confidence interval? I would bet that in your sample, the confidence interval for females includes small odds ratios, so any claim that there is no effect for females is invalid given the possibility of a large effect.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you - the intent is to identify subgroups driving the effects to propose future lines of study, so it's definitely on the descriptive end. $\endgroup$
    – lionclw
    Apr 4, 2022 at 18:53
  • $\begingroup$ Okay, just remember to not talk about significance and whether there is or isn't an effect. Just describe what you saw in the data. Also, odds ratios may not be the best way to describe the results since no one can interpret them; use risk ratios or risk differences instead. $\endgroup$
    – Noah
    Apr 4, 2022 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.