How to Calculate the Confidence Interval of the Sampling Mean with Scale Sensitivity and Varying Sample Sizes?

Question

I am currently facing a statistical challenge while comparing the average weights of two populations of larvae, and I need help figuring out how to perform the analysis correctly given the constraints of my data collection.

Scenario:

1. I have two large populations of larvae, each with unknown standard deviations.
2. I need to sample both populations and determine if the difference in the average weights between the two populations is statistically significant.
3. In a normal scenario, I would take two samples, calculate their means, and perform a t-test with unpooled variance (Welch's t-test) as per the Central Limit Theorem (CLT).

However, I cannot measure individual larval weights because the scale I’m using has a sensitivity of 100 micrograms, and the larvae themselves weigh much less than that. Therefore, I don’t have individual weights, and I cannot calculate the unpooled variance in a traditional manner.

What I Can Do: Instead of individual larval weights, I can take multiple large samples from both populations and record:

• The total weight of each sample.
• The size of each sample.
• The average weight per sample (calculated by dividing the total weight by the number of larvae).

The Problem: I’m looking for a method to compare the two populations using the limited data I have (total sample weights and average weights for varying sample sizes). There are a few complications:

1) Inference from observed Sampling Distribution:

Instead of estimating the variance of the sampling distribution through the individual observations within a single sample, I must estimate it using the variance of the observed sample means of multiple samples. There are multiple parameters to consider here - like the number of samples taken, as well as the individual sample sizes. How do I factor these into the calculation for the test?

2) Different Sample Sizes:

Each sample has a different size because the larvae are arbitrarily selected and weighed in the laboratory. The number of larvae is counted afterward using an ML computer vision model, which introduces variability in sample sizes. How should I handle these different sample sizes in the comparison?

3) Scale Sensitivity:

The scale rounds the total weights due to its sensitivity (100 micrograms). For example, if the total weight is recorded as 500 micrograms, the actual weight could range between 450 and 550 micrograms. We can assume that for a measured total weight X, the actual weight is uniformly distributed between 𝑋−50 and X+50. How do I account for this rounding error in the statistical test?

Note: Non-parametric tests are not an option here. I need to be able to detect effect sizes on a continuous scale, with a set confidence level and statistical power.

Given these challenges, how can I rigorously compare the two populations to determine whether the difference in average larval weights is statistically significant?

I appreciate any guidance on the appropriate statistical methods or adjustments I should make!

Thank you.

Answer 1

This is an interesting problem (+1), and is a good "real life" situation, where the measurement system is not as good as one would like (this happens a lot). So I will give it a try.

Caveat; a lot of the "recommendations" will involve a fair amount of hand waving. In such situations, there is no rigorous, formally correct solution; there are just "good enough" solutions. This is where engineers, like me, strive (and yes, as the old joke has it, we are happy enough with $\pi=3$).

The first thing I would do is perform a Gage Repeatibility and Reproducibility study (GR&R study) of your entire measurement system (operator + scale + machine vision), If not familiar with this, see here on wiki, or here; the "statistics" part is basically an ANOVA.. You need to make sure your gage (aka overall measurement system) introduces an acceptable amount of error into your measurements. A gage responsible for 10% or less of the overall variance of the results (which include "unit variance") is generally considered "good enough" (with ~1% considered "best in class").
It should be clear that if you put on the scale $\approx 100\mu g$ of larvae, your scale introduces $\approx 50\%$ error. But if you weigh $\approx 6000\mu g$ of larvae, this adds $\approx 1 \%$ error (but I would think your machine vision algorithm can not handle such a large count? $\approx 300$?). Same thing for the machine vision algorithm; how repeatable are the counts? $\pm 1 \%$, or $\pm 20\%$? What is the scale repeatibility (does it give you the exact same weight, for the same "bucket of larvae" each time?).

Yes, this is extra work, it is time consuming, etc... But if your measurement system is inadequate, no matter how clever your data analysis is, you will not escape the good old GIGO adage.

If your GR&R contribution to the total variance is below 10% (or 5%?), you are good to go. I would not worry about the scale precision (it averages itself out, with sufficient sample sizes). If not, then you have work to do.
What can you do? Increase the weight for each scale measurement (to diminish the impact of scale precision). But maybe the machine vision can not handle higher counts (overlap of objects etc.). Reduce the total weight of each measurement, to reduce the variability in machine vision readings (yes, catch-22...). Make repeated measurements (scale and/or machine vision), of the same "bucket", and average these to get at a better observed value. It depends what part of the measurement system is the "weak link", what is practical in your case, etc.

But in any case, I would want to make sure, before I invest a lot in the analysis (and its results!), that my measurement system was capable.

Assuming that it is, or that you fixed it so it becomes capable, there is no formal way to retrieve standard deviation from just mean and sample size (that should be clear; 2 samples with exactly the same mean and size could have wildly different sd's). So what can we do? A bit more hand-waving...
If the sample counts are "close" (single digit % off), just pretend they are all the same, use the formula for standard error, and derive $\sigma$ from it. Yes, it is not formally correct, but it may be good enough. It certasinly will not be a bad estimate.
Now note that if you made your GR&R acceptable, the counts should not be wildly variable; that alone is a problem you should fix. You can not make them exactly the same, but you should (need to?) be able to keep them in a narrow range.
Alternatively, depending on your data, while, for a given population, the sample sizes are different, you could have some observations where they are the same, or at least close enough (says he, while waving hands). Then use just these observations, to derive $\sigma$, while ignoring the other observations. Or collect more observations till you have "enough" of close ones, to approximate $\sigma$.
I know, this may all feel very unsatisfactory. But as Tukey said,

An approximate answer to the right question is worth far more than a precise answer to the wrong one.