2
$\begingroup$

I'm analyzing data from an experiment in which two independent groups were exposed to an experimental setup without and with treatment.

I am testing whether the treatment changed the second group's behaviour by performing a chi-squared test that compares group 2 (the observed) vs group 1 (the expected). The result indicates there is a significant change in behaviour X² p-value < 0.00014.

Now, I am trying to test subgroups to understand better the change, i.e., looking at gender, age, and other self reported metrics.

My question is, given that group 2 N=40 if I look at age for instance I find people in their 20s and their 60s show significant change but other age groups don't. However people in their 20s N=12 and people in their 60s N=5. Is there a heuristic / rule that says there is a minimum number of people needed to consider a result significant? For instance anything below N=5 cannot be considered significant or anything below N=20% of the population?

EDIT: Just to clarify, I am doing a chi-squared test of independence (between group 1&2) not a chi-square goodness of fit test.

EDIT 2: With this edit I consider the question closed. None of the answers / comments gave me a definitive solution, which I believe says more about the question than the answers. I was hoping for a definitive answer along the lines you need at least 5 people or 20% of your sample. It seems the answer is less direct as it is sensitive to many factors.

Improve this question
$\endgroup$
5
  • $\begingroup$ I don't understand what you mean by "expected" -- I thought that group 1 was also collected data, and not theoretical distribution? $\endgroup$
    – January
    Commented Nov 16, 2012 at 8:52
  • $\begingroup$ Group one is not exposed to treatment but yes to environment so their behaviour is what is expected when the treatment is not present in a given environment. Once you introduce the treatment then you compare the behaviour in the same environment pre and post treatment. The resulting change is the observed effect of treatment in that environment. This is standard procedure in HCI/social sciences research. $\endgroup$
    – G Garcia
    Commented Nov 16, 2012 at 11:26
  • $\begingroup$ You are going to confuse a lot of people if you use "expected" in that way, as it means something else when talking about chi-square tests. $\endgroup$
    – Peter Flom
    Commented Nov 16, 2012 at 11:54
  • $\begingroup$ Why are you using chi-square at all? You have some dependent variable, apparently dichotomous (although you haven't said) and several independent variables (treatment, age, gender etc). That calls for regression of some sort, probably logistic regression if I am right about the DV. $\endgroup$
    – Peter Flom
    Commented Nov 16, 2012 at 11:58
  • $\begingroup$ I'm doing chi-square because I need to look at each variable in isolation not as a group, in which case I would probably do an ANOVA. $\endgroup$
    – G Garcia
    Commented Nov 16, 2012 at 14:31

1 Answer 1

Reset to default
2
$\begingroup$

For small sample sizes, use Fisher's exact test, because the $\chi^2$ test sampling statistics has only approximately the $\chi^2$ distribution, and this approximation is problematic for small sample sizes.

While lower sample size decreases the power of the test, the p-values (and not the sample size) are indicators of the statistical significance. A significant p-value stays significant whatever the sample size; the sample size has been taken care of through the calculation of the test statistic.

However, someone might claim that a small sample size is more likely to be biased. This is not necessarily true, but I think there might exist a correlation between the study sample size and whether the data was collected in an unbiased way as it should.

Improve this answer
$\endgroup$
3
  • $\begingroup$ My understanding is that Fisher tests for the independence of variables on a contingency table. However I want to compare the observed against the expected not between two observed measures. Fisher could tell me how independent the variables are, but that whould not shed any light on whether the treatment had any effect when compared to previous behaviour. Am I correct on my interpretation? I know p-value is the indicator of significance, my question is if p-value is significant but sample is a small proportion of the population, does that invalidate the significance? $\endgroup$
    – G Garcia
    Commented Nov 16, 2012 at 8:10
  • $\begingroup$ You are not doing a goodness-of-fit here, I thought you are comparing two populations. I read your question again, and I think that I am confusing about what you are doing. Can you please elaborate on what you are comparing? Regarding the sample size: see updated answer. $\endgroup$
    – January
    Commented Nov 16, 2012 at 8:46
  • $\begingroup$ I am comparing group1 (my null case, i.e no treatment) vs group2 (my group exposed to treatment). So comparing two contingency tables as opposed to columns within one. $\endgroup$
    – G Garcia
    Commented Nov 16, 2012 at 9:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.