I would like advice on which test is appropriate for the problem below. I have read numerous posts about hypothesis tests for small sample, non-parametric tests, testing differences in proportions and am completely confused. Please can anyone provide guidance- Thanks – Details below

My dataset: My sample consists of 47 companies, which fall in two groups. Group 1 (G1) consist of 12 companies and Group 2 (G2) consists of 35 companies. I have data on these companies for 3 years (balanced panel 141 observations). I have 25 indicators which relate to actions that these companies have undertaken. These indicators are dichotomous, so if a company undertakes an action (e.g. on indicator1) it is scored 1 and 0 otherwise. For each of the 25 indicators, I have counted and calculated the number and percentage of group1 and group2 companies undertaking the action (i.e. the number and percentage of companies which score 1 for each indicator) in each year. So I have a table of data which looks like

Year1: G1: 9 (75%) G2: 22(63%)

Year2: G1: 7(58%) G2: 21(60%)

Year3: G1: 10(83%) G2: 22(63%)

Average: G1: 9(72%) G2: 22(62%)


Year1: G1: 0 (0%) G2: 2(6%)

Year2: G1: 0(0%) G2: 2(6%)

Year3: G1: 1 (8%) G2: 1(3%)

Average: G1: 0(3%) G2: 2(5%)


Year1: G1: 8(67%) G2: 28(80%)

Year2: G1: 7(58%) G2:27(77%)

Year3: G1: 7(58%) G2:27(77%)

Average: G1: 7(61%) G2:27(78%)


Year1: G1: 3 (25%) G2: 0(0%)

Year2: G1: 3 (25%) G2: 0(0%)

Year3: G1: 3 (25%) G2: 0(0%)

Average: G1: 3 (25%) G2: 0(0%)

Where e.g. 9 is the number of group1 companies undertaking action on indicator1 which is 75% of the group1 companies.

Problem: I would like to test for each indicator: if the difference in the average number (or percentage) of G1 and G2 companies is statistically significant. i.e. for indicator 1 is 9(72%) statistically significantly different from 22 (62%). Am I right in even trying to test this, given that 1) the average represents the average no’ (percentage) of companies across the 3 years and 2) on some indicators there are such few number of companies undertaking the actions e.g. see indicator 2

A colleague of mine recommended non parametric testing specifically Kruskal Wallis test but did not explain why. Any advice on above is appreciated. Finally apologies if the question is not in the correct format as required on this website, as this is my first time posting.


1 Answer 1


Would it not work to do a 2x2 chi-square test for each indicator? That would test for independence between group membership and a "yes" or "no" on implementation of the indicator. If your result from the chi-square was significant, you would reject the null hypothesis that group membership and indicator are independent. I am open to being corrected, but as far as I am aware, that would be an acceptable way to do it.

As another point, relating to your sub-question about whether using averages of the three years is appropriate. I think that may be problematic because you are using time-series data but are not accounting for the temporal relationship in any way. As an alternative, you could use a cumulative measure: i.e. instead of looking at the average of yes/no through the three years, you could take the percentage of companies for which the answer was "yes" in at least one year. That would allow you to side-step the temporal angle in a way that I believe would still be justifiable.

  • $\begingroup$ Thanks for your answer. And the advice regarding the use of cumulative measure. Just a quick question about the chi sq. Test. I have read that for ch sq. Test the no' of successes (yes) and failures (no) need to be at least 5 in each group of companies which does not hold for all 25 indicators (see example in question). In this case I have read that exact methods like Fisher exact test should be used- can I / should I use chi sq where appropriate and fisher's exact test where <5 successes/ failures? Thanks $\endgroup$
    – SNAV
    Commented Nov 6, 2015 at 10:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.