6
$\begingroup$

I'm comparing two groups of people on a binary variable. The number of successes is really small for the two groups (which wasn't expected, due to a lack of previous quantitative research on the subject) and the number of total observations is not so great (450), so we have group A with 2 successes on 225 observations, and group B with 10 successes on 225 observations.

The p-value from a chi-squared test is significant but really close to the pre-set alpha level (the significance level was 0.05 and the p-value is about 0.04). A problem I see is that for example just one more success in group A would make the test non significant.

For these two reasons (small numbers + p-value really close to the significance level), I have serious doubts about the "significant" result being in any way reliable.

Are there some good references that can confirm (or infirm, for that matter) my intuition, and that I could refer my readers to as a form of caveat?

I still want to report the p-value but attach a caveat to it, to show to interested people that a larger sample size would be required to reproduce the study and get a more reliable result. There's some qualitative research data hinting to a difference between the two groups, but for the moment there's a lack of reliable statistical/quantitative research that would show the same thing or contradict it.

Thanks for any pointer!

Improve this question
New contributor
Coris is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
1
  • 3
    $\begingroup$ Congratulations on how you're handling this. Perhaps not quite what you're looking for, but Chapter 4 of Frank Harrell's text on Regression Modeling Strategies notes that with a binary outcome you should be reluctant to fit any model where the number of cases in the minority class (12 in your case) is less than 15 times the number of predictors (1 predictor in your case, requiring 15 minority-class members). His online text puts it a bit differently, but gives the same "effective sample size" of 12. $\endgroup$
    – EdM
    May 25 at 15:51

3 Answers 3

Reset to default
6
$\begingroup$

My suggestion is to complement the p-value with a report of the effect size and its confidence interval (and don't use the word "significant"). For your data, the relevant effect size is the relative risk.

The relative risk is 5.0 and its 95% CI ranges from 1.3 to 20.2, from a 30% increase to a 20-fold increase. That wide range expresses the uncertainty you want to convey.

There are several ways to compute (estimate) the CI. I used Koopman asymptotic score implemented in GraphPad Prism. Here is a screenshot of the results: enter image description here

Improve this answer
$\endgroup$
6
$\begingroup$

This is a nice example of a situation where the all too conventional all-or-none description of the results as 'significant' or 'not significant' is unhelpful, as you have sensibly noticed. I agree with Harvey Motulsky's answer, but would add a few considerations.

First, note that a p-value of 0.04 is only ever fairly weak evidence against the null hypothesis. Where your the sample size is small the false positive error rate does not increase (assuming the statistical model is appropriate), but a 'significant' result will often come from a sample that severely exaggerates the true effect.

Next, note that the individual binary data points carry relatively little information, and so even though the p-value falls within the conventionally 'significant' range, the likelihood function from your data will be relatively widely spread and so if you were to do a Bayesian analysis (maybe you should!) you would find that the posterior probability distribution would not be moved very much from your prior. (All of that is to say that a a result of p=0.04 is says that you have fairly weak evidence against the null.)

Finally, note that the reliability of a statistical test is only as good as the statistical model is well matched to the actual data generating and sampling systems.

Given your thoughtful question, you might find my explanations of the differences between p-value evidence considerations and error rate accountings will clarify the issues. You will find it interesting in any case: A reckless guide to p-values: local evidence, global errors.

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ "so if you were to do a Bayesian analysis (maybe you should!) you would find that the posterior probability distribution would not be moved very much from your prior" - @Coris stated that the small number of successes was a surprise, so the posterior may very well be different from an honest prior. Setting up an honest prior however is very hard and maybe not credible after the results are already seen. $\endgroup$
    – Christian Hennig
    May 25 at 23:55
  • 2
    $\begingroup$ @ChristianHennig Good point. Even weak evidence can move the posterior if it points far from a prior. $\endgroup$
    – Michael Lew
    May 26 at 1:31
  • 1
    $\begingroup$ Unfortunately I can't accept multiple answers, and I can't give a positive vote, so here is this short comment just to say thank you for your useful answer. I note the caveat by Christian Hennig too, but for future studies similar to this one, I think I'll consider Bayesian analysis as you suggest. $\endgroup$
    – Coris
    May 26 at 11:24
2
$\begingroup$

Great question.

I think the previous answers by @HarveyMotulski and @MichaelLew are solid: you set out to investigate a between-group difference, with positive but weak results, and especially given the replication crisis need to emphasize that while by your decision criterion this counts as a group difference, those results depend on a single observation.

Their answers concentrate on describing the experiment you set out to do, and the analysis you would have pre-registered. And I agree you have a duty to report that as planned, to help stave off file drawer bias and other scientific gremlins.

However, the key result to me is your surprise at the few successes. This may not be Fleming's petri dish, or Rutherford's reflected alpha particles, but it's a potential discovery. It seems you had a very important but unarticulated background assumption that only became apparent when it was violated.

Worth mentioning for its own sake and because that violation might invalidate the statistical test. I'm thinking here of Feynman's story about rats running mazes.

Gelman frequently emphasizes "check the fit" as a step that might be outside your formal inference framework: one example here. I think this is also consilient with Mayo's severe testing approach -- but I'm no expert.

That may lead you to a new model or explanation -- it would be exploratory work on this dataset and require another experiment to test, which hopefully you have time and resources for, else a good description can inspire someone else to.

Best of luck!

Improve this answer
$\endgroup$

Your Answer

Coris is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.