How to determine that sample population defect rate represents total population defect rate when all quantities are known

Question

I don't know how to correctly phrase this question, so sorry in advance.

Let's say I have data on two populations. Population 1 represents all of an item I've manufactured, and each item has the ability to tell me if it has a defect (using attribute data, item either has a defect or does not, and each item can only present one defect). Therefore, I have a total sample size and all the defect counts of my entire population. I made a change to this item, and I would like to know if this change affects my defect rate. I did some trials with the modified item using a smaller sample size, and I have a count of all the defects that occurred in this smaller sample size.

Population 1: 100000
Defect A Count: 400, 0.4%
Defect B Count: 8, 0.008%
Defect C Count: 0, 0%

Population 2: 500
Defect A Count: 2, 0.4%
Defect B Count: 1, 0.2%
Defect C Count: 0, 0%

How do I determine if the defect rates between the two populations are similar, and with how much confidence I can say that they are similar in this scenario? For Defect A, sure, they have the same defect rate, but the defect count for Defect A in population 2 is very small, so maybe this is not statistically significant.

Answer 1

First, do not worry too much whether the first dataset is the total population (to date...). You can still treat it as a sample from a "super-population" (see e.g. here on CV). This would include the units to be manufactured in the future, or which could have been manufactured in the past if only you had had the orders, etc...
So you can compare the 2 datasets, as if they are both samples (2 proportions tests). Or you can assume that the proportions from the first dataset are "the true proportions", and compare the 2nd proportions to this "truth" (but is it really the truth? what if you had manufactured a few more, or a few less?).
A 2nd issue is that failing to prove that they are different does not prove that they are the same. Take your "defect A"; both show a proportion of 0.4%. So are they the same? No; the 95% confidence binomial CI for the 2nd population goes from .05% to 1.44%: ~10 times less to ~3 times more than "the truth". Yes, 0.4% is inside the CI, but that does not prove anything.
What you really would need is an equivalence test (or probably better, a non inferiority test, if all you want to show is that the defect rates are not "much worse"). The real problem is that your 2nd sample is much too small, with way too few instances of defects to reach any conclusion. There is no chance to pass any such equivalence/non-inferiority test.
So whether you treat the first dataset as "the truth", or as a 2nd sample (since the sample size is rather large, the CI's are narrow: e.g. for defect A, the CI is .36% to .44%), will not make a difference, as your 2nd sample is too small, given the very low defect rates. This is a very common situation in industrial, manufacturing, engineering applications; we want very low defect rates (possibly 0), so proving that a change is an improvement (superiority testing), or not a practically significant change (equivalence test), or even not a practically significant deterioration (non-inferiority test), requires very large sample sizes, sometimes beyond what is practical/affordable.