Sample size question (small vs large)

Question

I have a question about sample sizes.

I have a data set in which I have following conditions:

1) Day 1: Total=3500, Fail=200;Fail%=6

2) Day 2: Total=6700, Fail=340, Fail%=5

3) Day 3: Total=1100, Fail=50; Fail%=5

4) Day 4: Total=500, Fail=17;Fail%=3

5) Day 5: Total=70, Fail=2;Fail%=3

Is there any way to know if Day 5 data can even be considered in this analysis since sample size is extremely small as compared to Day 1 sample size data (both pass and fails)?

Which statistics test to use for such problem?

Do I need more sample size on days 3,4 and 5 or can I just use the sample size which I have and say confidently that the Fail% decreases from Day 1 to Day 5?

Answer 1

You are right about the test of homogeneity. (I am a little surprised it's not significant, but that's just an argument for actually doing the test instead of trying to guess.) My results from Minitab 17 are as follows:

Chi-Square Test for Association: Result, Day 

Rows: Result   Columns: Day

            D1      D2      D3     D4    D5    All

Pass      3300    6360    1050    483    68  11261
        3320.4  6356.3  1043.6  474.3  66.4

Fail       200     340      50     17     2    609
         179.6   343.7    56.4   25.7   3.6

All       3500    6700    1100    500    70  11870

Cell Contents:      Count
                    Expected count

Pearson Chi-Square = 7.087, DF = 4, P-Value = 0.131
Likelihood Ratio Chi-Square = 7.566, DF = 4, P-Value = 0.109

* NOTE * 1 cell with expected count less than 5

[While there is no technical difficulty running the test with such different sample sizes, it is worthwhile noting that the power of the test (ability to detect if a null hypothesis is false) tends to depend heavily on the size of the smaller sample. Here the small sample size for Day 5 has also led to a small expected cell count for 'Fail on Day 5'. Ordinarily, one such small count out of 10, still above 3, would not be taken to invalidate the approximation of the test statistic to the chi-squared distribution.]

Also, if I do a test to compare Day 1 with Day 5, I don't get a significant result, perhaps because of the relatively small sample for Day 5.

Test and CI for Two Proportions 

Sample    X     N  Sample p
1       200  3500  0.057143
2         2    70  0.028571

Difference = p (1) - p (2)
Estimate for difference:  0.0285714
95% CI for difference:  (-0.0112064, 0.0683493)
Test for difference = 0 (vs ≠ 0):  Z = 1.41  P-Value = 0.159

* NOTE * The normal approximation may be inaccurate for small samples.

Fisher’s exact test: P-Value = 0.435

However, if I do a test to compare Days 1 and 4, I get a result significant at the 1% level (with normal approximation, or 3% level with Fisher's Exact test):

Test and CI for Two Proportions 

Sample    X     N  Sample p
1       200  3500  0.057143
2        17   500  0.034000

Difference = p (1) - p (2)
Estimate for difference:  0.0231429
95% CI for difference:  (0.00549430, 0.0407914)
Test for difference = 0 (vs ≠ 0):  Z = 2.57  P-Value = 0.010

Fisher’s exact test: P-Value = 0.034

Finally, if I compare the first three days with the last 2, I get a significant result.

Test and CI for Two Proportions 

Sample    X      N  Sample p
1       570  10710  0.053221
2        19    531  0.035782

Difference = p (1) - p (2)
Estimate for difference:  0.0174397
95% CI for difference:  (0.00107916, 0.0338003)
Test for difference = 0 (vs ≠ 0):  Z = 2.09  P-Value = 0.037

Fisher’s exact test: P-Value = 0.089

Notice that we have several tests now. It would have been best to decide at the start (before seeing data) which test we would rely on to the give the most accurate view of the results.

If I were to show the test just above (First three vs. Last 2) in a report of findings, I would feel obligated to mention in a brief footnote which other tests did not give significant results. And, perhaps mentioning the Fisher P-value, I would claim a 'suggestive' finding rather than a 'convincing' one.

Answer 2

Another option would be to compute a logistic regression using fail (or pass) as the dependent variable and day as the independent (covariate) variable.

Computing a logistic regression will give you a coefficient $B=-0.149$, $p=0.009$ (I used SPSS 25.0). Since $e^B=0.862$, it means that, on average, each day that passes will have a lower odds of failing, with a daily decrease of around $13.8\% = 100\% \times (1 - 0.862)$. So, on day $d$, the odds will be smaller than the odds of the previous day, and be given approximately by $odds_d = 0.862 \times odds_{d-1}$.

It might be interesting to also look at a related question What does “linear-by-linear association” in SPSS mean? Using the linear-by-linear association also results in $p=0.009$.