I'm comparing two groups of people on a binary variable. The number of successes is really small for the two groups (which wasn't expected, due to a lack of previous quantitative research on the subject) and the number of total observations is not so great (450), so we have group A with 2 successes on 225 observations, and group B with 10 successes on 225 observations.
The p-value from a chi-squared test is significant but really close to the pre-set alpha level (the significance level was 0.05 and the p-value is about 0.04). A problem I see is that for example just one more success in group A would make the test non significant.
For these two reasons (small numbers + p-value really close to the significance level), I have serious doubts about the "significant" result being in any way reliable.
Are there some good references that can confirm (or infirm, for that matter) my intuition, and that I could refer my readers to as a form of caveat?
I still want to report the p-value but attach a caveat to it, to show to interested people that a larger sample size would be required to reproduce the study and get a more reliable result. There's some qualitative research data hinting to a difference between the two groups, but for the moment there's a lack of reliable statistical/quantitative research that would show the same thing or contradict it.
Thanks for any pointer!