How to check if removing a sample makes a difference in mean and stdev values?

Question

I'd like to ask if someone could help me with the following problem:

we have measured the same sample 5 times and we would like to check if there are significant differences in mean and stdev values if we use:

• All 5 datapoints
• Only the last 4 datapoints
• Only the last 3 datapoints

We have performed ANOVA analysis but we are not sure about the results because we might not have homocedasticity.

Which tests would you do to investigate this issue?

Thanks in advance for your help.

Answer 1

Not only don't you have much data (it's very unlikely to be enough to pick up any change even moving from 5 points to 3, even if the effects are large). The things you propose comparing with each other and the sequence of tests you propose are also dependent.

With appropriate assumptions, even at very small sample sizes you can make some such comparisons, but the power will be very low, and your dependence on the particular assumptions may be the major factor in your conclusions.

Answer 2

You can consider some kind of jackknife approach in which each reading is compared against the mean of all other runs except the reading being contrasted.

For example, in one of your comments you mentioned a set of number:

$62.970, 62.202, 61.791, 56.588, 59.155$

To contrast the first one with the rest:

1. Calculate the means of $62.202, 61.791, 56.588, 59.155$, which is $59.934$.
2. Find out the percent difference: $(62.970 - 59.934)/59.934 \times 100\%$, which is $5.07\%$
3. Repeat this for the rest, the whole data should be: $5.07\%, 3.45\%, 2.59\%, -8.03\%, -2.85\%$

Now, this tells a lot of problems. First your assumption that readings get more consistent is challenged here. The "odd-one-out" is actually the fourth run. Another worrying sign is the consistent autocorrelation. A consistent test shouldn't have all the positive error in the front, and negative error at the back, aka, the positive and negative should scatter.

This brings back to your original question. I'd suggest that statistics isn't the core problem here, but the quality control of your device. If "unclean" or "non-stablized" is your concern, then please check the system manual and investigate how to properly rinse your coil, and how to use test run reagent to estimate the efficiency. If you come back to judge your readings in such a post-hoc manner, "weeding" out weird data, your results can invite substantial suspicion.

In a nut shell, I'm not exactly a lab-technician, but from a general researcher's point of view all these data cleaning criteria should be set a prior, that is before you have gotten any data. Once you have the data, and the protocol was followed, and there wasn't detected problem, then the number should be analyzed.

Answer 3

Given a data set $\{y_1, y_2, \ldots, y_n\}$ with $n$ entries, the mean $\mu$ satisfies $n\mu = \sum_{i=1}^n y_i, $ while the mean of the expurgated set $\{y_1, y_2, \ldots, y_{n-1}\}$ is $$\hat{\mu} = \frac{1}{n-1}\sum_{i=1}^{n-1} y_i = \left.\left.\frac{1}{n-1}\right(n\mu - y_n\right)$$ which will equal $\mu$ exactly if and only if the deleted entry $y_n$ equals $\mu$. Thus, deleting an entry from a data set will change the mean unless the point deleted happens to equal the mean of the original data set.

The variance $\sigma^2$ of the original data set satisfies $(n-1)\sigma^2 = \sum_{i=1}^{n}(y_i-\mu)^2$. If we delete an entry (say $y_n$) which happens to have value $\mu$ (so that the mean remains the same), then the expurgated data set has variance $$\begin{align} \hat{\sigma}^2 &= \frac{1}{n-2}\sum_{i=1}^{n-1} (y_i-\mu)^2\\ &= \frac{1}{n-2}\sum_{i=1}^{n} (y_i-\mu)^2 &\text{since}~ y_n-\mu = 0,\\ &= \frac{n-1}{n-2}\sigma^2\\ &> \sigma^2. \end{align}$$ Thus, our effort to preserve the mean necessarily increases the variance.

Answer 4

Comments:

• Five samples is pretty low. I prefer 98, or 300, or tens of thousands.
• As I am understanding the question, it seems pretty simple to answer using Excel using brute methods.
• Really the number of required samples is governed by the function that you are trying to measure. Metropolis-Hastings often takes tens of thousands to get "its feet on the ground".

Answer:

You can use bootstrap resampling to investigate the uncertainty in both the mean and the standard deviation for different sample counts. This will use the data you have, not the date you wish you had, and give you an answer that is about as meaningful as possible. So after you find say 300 estimates of the mean with uniform random sampling with replacement (bootstrapping) then you can perform EDA on those values by graphing the CDF and looking at both central tendency and variation.

I do not feel comfortable taking thousands of resamples when I only have 5 actual measurements. Even 300 is a bit high, but it is at an boundary where results get to be clean and still remain informative.