2
$\begingroup$

I have a question about sample sizes.

I have a data set in which I have following conditions:

1) Day 1: Total=3500, Fail=200;Fail%=6

2) Day 2: Total=6700, Fail=340, Fail%=5

3) Day 3: Total=1100, Fail=50; Fail%=5

4) Day 4: Total=500, Fail=17;Fail%=3

5) Day 5: Total=70, Fail=2;Fail%=3

Is there any way to know if Day 5 data can even be considered in this analysis since sample size is extremely small as compared to Day 1 sample size data (both pass and fails)?

Which statistics test to use for such problem?

Do I need more sample size on days 3,4 and 5 or can I just use the sample size which I have and say confidently that the Fail% decreases from Day 1 to Day 5?

$\endgroup$
  • 1
    $\begingroup$ What do you want to know? Perhaps first test whether the percentage failing might not be significantly different from day to day; then if truly different, specifically compare Days 1 and 5? // Why are sample sizes smaller on later days? Is that an important part of the story in itself? Are 'participants' randomly chosen and different on different days? // For most 2-sample tests, having unbalanced sample sizes is not a deal-breaking difficulty. (You could ask whether 6700 is too many as well as whether 70 is too few.) $\endgroup$ – BruceET Feb 28 '19 at 1:08
  • 1
    $\begingroup$ @BruceET, sample sizes are different since the process I am working on is days dependent. When day is 1, more samples exist vs when day is 5, very less samples exist. This is an experimental data. We did not choose it to be this way. Maximum time between two processes is 5 days. It is just that more samples exist on day 1 vs on day 5. All sample sizes are randomly chosen. It is like an expiry date kind of. Expiry date is 5 days. So we can sell on day 1 to day 5 itself. This is just a related example. $\endgroup$ – excelislife Feb 28 '19 at 1:11
  • $\begingroup$ Thanks for the reply. Can you please explain in more details @BruceET? $\endgroup$ – excelislife Feb 28 '19 at 1:47
  • 1
    $\begingroup$ On using chi-sq test for homogeneity, I got p value of 0.13 and chi square of around 7. $\endgroup$ – excelislife Feb 28 '19 at 3:20
1
$\begingroup$

You are right about the test of homogeneity. (I am a little surprised it's not significant, but that's just an argument for actually doing the test instead of trying to guess.) My results from Minitab 17 are as follows:

Chi-Square Test for Association: Result, Day 

Rows: Result   Columns: Day

            D1      D2      D3     D4    D5    All

Pass      3300    6360    1050    483    68  11261
        3320.4  6356.3  1043.6  474.3  66.4

Fail       200     340      50     17     2    609
         179.6   343.7    56.4   25.7   3.6

All       3500    6700    1100    500    70  11870

Cell Contents:      Count
                    Expected count

Pearson Chi-Square = 7.087, DF = 4, P-Value = 0.131
Likelihood Ratio Chi-Square = 7.566, DF = 4, P-Value = 0.109

* NOTE * 1 cell with expected count less than 5

[While there is no technical difficulty running the test with such different sample sizes, it is worthwhile noting that the power of the test (ability to detect if a null hypothesis is false) tends to depend heavily on the size of the smaller sample. Here the small sample size for Day 5 has also led to a small expected cell count for 'Fail on Day 5'. Ordinarily, one such small count out of 10, still above 3, would not be taken to invalidate the approximation of the test statistic to the chi-squared distribution.]

Also, if I do a test to compare Day 1 with Day 5, I don't get a significant result, perhaps because of the relatively small sample for Day 5.

Test and CI for Two Proportions 

Sample    X     N  Sample p
1       200  3500  0.057143
2         2    70  0.028571

Difference = p (1) - p (2)
Estimate for difference:  0.0285714
95% CI for difference:  (-0.0112064, 0.0683493)
Test for difference = 0 (vs ≠ 0):  Z = 1.41  P-Value = 0.159

* NOTE * The normal approximation may be inaccurate for small samples.

Fisher’s exact test: P-Value = 0.435

However, if I do a test to compare Days 1 and 4, I get a result significant at the 1% level (with normal approximation, or 3% level with Fisher's Exact test):

Test and CI for Two Proportions 

Sample    X     N  Sample p
1       200  3500  0.057143
2        17   500  0.034000

Difference = p (1) - p (2)
Estimate for difference:  0.0231429
95% CI for difference:  (0.00549430, 0.0407914)
Test for difference = 0 (vs ≠ 0):  Z = 2.57  P-Value = 0.010

Fisher’s exact test: P-Value = 0.034

Finally, if I compare the first three days with the last 2, I get a significant result.

Test and CI for Two Proportions 

Sample    X      N  Sample p
1       570  10710  0.053221
2        19    531  0.035782

Difference = p (1) - p (2)
Estimate for difference:  0.0174397
95% CI for difference:  (0.00107916, 0.0338003)
Test for difference = 0 (vs ≠ 0):  Z = 2.09  P-Value = 0.037

Fisher’s exact test: P-Value = 0.089

Notice that we have several tests now. It would have been best to decide at the start (before seeing data) which test we would rely on to the give the most accurate view of the results.

If I were to show the test just above (First three vs. Last 2) in a report of findings, I would feel obligated to mention in a brief footnote which other tests did not give significant results. And, perhaps mentioning the Fisher P-value, I would claim a 'suggestive' finding rather than a 'convincing' one.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks @BruceET, it was an extremely detailed and clear reply! $\endgroup$ – excelislife Mar 2 '19 at 9:59
0
$\begingroup$

Another option would be to compute a logistic regression using fail (or pass) as the dependent variable and day as the independent (covariate) variable.

Computing a logistic regression will give you a coefficient $B=-0.149$, $p=0.009$ (I used SPSS 25.0). Since $e^B=0.862$, it means that, on average, each day that passes will have a lower odds of failing, with a daily decrease of around $13.8\% = 100\% \times (1 - 0.862)$. So, on day $d$, the odds will be smaller than the odds of the previous day, and be given approximately by $odds_d = 0.862 \times odds_{d-1}$.

It might be interesting to also look at a related question What does “linear-by-linear association” in SPSS mean? Using the linear-by-linear association also results in $p=0.009$.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.