Testing equivalence for consistency validation

Question

I am testing the output of a low-power cell battery and then I measured the output after a while. I did multiple runs and the output was slightly different every time. I would like to check whether my collected data are aligned with the population and show that my output measurements are reliable and consistent. The standard is to show a P-value of P < 0.05.

My approach (MATLAB):

mu = 2.366;     % Population mean 
sample=[2.180213,2.178298   ,2.310851   ,2.114255   ,3.012553   ,2.69234    ,2.079787];
n = numel(sample);
[h,ptest] = ttest(sample,mu,0.05,'right')

(NOTE: all values are in micro *10^-6)

I am getting a value of p = 0.49 & h = 0. h = 0 indicates that ttest does not reject the null hypothesis at the 5% significance level.

Is there something I am doing wrong or are those the right p & h values for my output? can I reduce the sensitivity for my test to get better results?

Answer 1

I think that you are carrying out the wrong test for your real hypothesis. From your comments, you want to show that your numbers are "close together" and not "far apart". This is a statement about the standard deviation, not the mean. Your test right now can only tell you about the mean and not about the standard deviation. Let me give you two examples:

2.369 2.384 2.373 2.361 2.362 2.356 2.356 (mean = 2.366, standard deviation = 0.01)

4.478 0.358 3.375 3.559 1.632 2.430 0.730 (mean = 2.366, standard deviation = 1.54)

I'm guessing that you would find the first sample "acceptably close together" and the second "too far apart". If this is right, you'll need to do a chi-square test for standard deviation.

Your next steps are:

1. Decide how large a standard deviation is acceptable to you (0.01? 0.1? 0.5?). Note that you CANNOT get this from your sample data. It needs to be a number that you choose for reasons other than the data you collected.
2. Carry out a one-sided chi-square test with H0: standard deviation >= X vs HA: standard deviation < X, where X is the largest standard deviation you would be happy with.
3. If you reject the null hypothesis with p < 0.05, you can say your battery power outputs are sufficiently consistent for you.

Unfortunately I am not familiar with MATLAB and so can't implement step 2, but hopefully this gives you a better idea on what to search for.

Answer 2

To answer the first part of your question, i.e. what you are doing wrong. In your example, you are testing the following hypotheses:

$H_0: \mu_s \le \mu$ and $H_1: \mu_s > \mu$ where $\mu = 2.366$

In other words, you are trying to accept the alternative hypothesis that states that the true mean is greater than 2.366... but you failed to reject the null hypothesis (which states that the true mean is less or equal to 2.366)!

EDIT FOR FUTURE READERS INTERESTED BY THIS QUESTION

Comments on (accepted) answer proposed by @user334895: testing variance does not lead to equivalence. The collected information could have a different mean but a similar variance and, therefore, lead to an incorrect conclusion.

What is implied in this answer is that if the mean of the obtained sample is very close to the population mean, then we only need to test the variance of the sample to conclude to equivalence. The problem is that it is actually a mix of descriptive statistics (use of the sample mean and assume it is equal the true population mean) and inferential statistics (testing variance to draw conclusion about the population variance).

However, the obtained sample is one possible out of many others, so there is no certainty about its mean. Based on the sample we have, we can empirically illustrate/simulate this sampling distribution of the sample means with bootstrap:

A mean of 2.6 is also possible for example, so the variance test is no longer valid in this scenario because even if variance matches the target, the data collected will differ from the population mean.

An approach to solve this problem would be to run a TOST (Two-One-Sided T-test) procedure (equivalence testing). It is already well documented on this site https://stats.stackexchange.com/a/500121/321901.