1
$\begingroup$

Summary: We have both an old tried and true computer (type #1) and brand new (type #2) on 100 airplanes processing the same raw data and want to know if they are equivalent based on the telematics output of those computers. Should we use the sample size equation for two independent samples continuous outcome

$$n_i = 2\left(\frac{Z\sigma}{ES}\right)^2$$

or sample size for matched continuous outcome

$$n = \left(\frac{Z\sigma_d}{E}\right)^2$$

Details:

We have a fleet of airplanes with onboard computers (type #1) that process raw telematics data from hardware such as the engine, brakes, GPS, altitude, etc. We are testing a new onboard computer (type #2) and hypothesize type 2’s will process that raw data identically to type #1. We run a 100 plane experiment where both type 1 and type 2 computers are present on each of the 100 planes and process the same raw data. We have flown 1 million miles so far and our CEO wants to know the sample size needed to decide if the devices are similar enough, e.g. accept or reject the null hypothesis.

What I’ve tried:

We started by using the sample size for two independent samples where the standard deviation is derived from the known standard deviation of our Type 1 computer. However, now I believe that it would be more appropriate if we used the matched sample equation for sample size because we had 100 planes with both type 1 and a type 2 computer onboard and the output are not truly independent but fit the matched testing scenario better. Since both type 1 and type 2 are present on each plane and we are comparing the outputs of type 1 vs type 2 per plane I think we need the sample size for matched samples equation where the standard deviation is based on the difference between type 1 and type 2 that we have observed so far based on the 1 million miles flown so far with both type 1 and type 2. Are both sample size equations acceptable, or do we have to pick one, and why, please? Thank you!

$\endgroup$
4
  • 1
    $\begingroup$ Let's back up here. How are you measuring "processing". For each experimental unit (planes I suppose), what is the outcome of the measurement processes? Is it a number? Is it continuous? Let's start there. $\endgroup$ Commented Sep 18, 2021 at 18:43
  • $\begingroup$ So a concrete example of “processing” for both type 1 and type 2 computers: they are logging speed of the plane every second based on the gps sensor and both outputting speed every second. This output well call a parameter is similarly outputted every second for other parameters like engine rpm, fuel usage, etc. Every second we get a continuous value from both computers, and I’m tasked with determining if the type 2 computer parameter output is similar to the tried and true type 1 parameter output $\endgroup$ Commented Sep 18, 2021 at 18:56
  • $\begingroup$ Instead of “processing” I should have rather said “logging”. The two computers are logging sensor readings every second. I am calculating sample size on each parameter continuous value, and the parameter that requires the highest sample size will in a way dictate how many samples we need overall since each parameter is logged at a fixed rate of once per second $\endgroup$ Commented Sep 18, 2021 at 19:23
  • $\begingroup$ My question is that since the two computers reside on the same plane wouldn’t this be an example of a dependent matched pair (e.g. control and treatment on the same subject) as opposed to independent groups? That classification helps me decide on the appropriate sample size formula. $\endgroup$ Commented Sep 19, 2021 at 0:39

2 Answers 2

0
$\begingroup$

Let me take a step back and try to frame the problem into some existing theory; would love to get a statistician's take on it, too.

It sounds like the objective of the analysis is to consider $k$ parameters, each of which is measured many times by two, different, computers $c$ during the course of a flight, giving $n_p$ samples for each computer and parameter combination, where $p = 1..k$ and we have $n_p$ samples from each computer, across 100 airplanes ($a = 1..100$). The measurements for a given combination of parameter, computer and plane are time series, will not be independent and identically distributed (subsequent measurements are not statistically independent of previous measurements, nor are they sampling from the same distribution because, e.g., position and speed are always changing), but we expect measurements between the two computers to be highly correlated and, in fact, we'll try to test if the error (difference) between the computers is zero.

I'll assume that the two computers take measurements at the same times and that, for each parameter, the errors between them at each point in time are independent and identically distributed both across the planes (error between computers on one plane is not a function of error between computers on another plane, and, in fact, comes from the same distribution for every plane) as well as over time (error between computers is not a function of what point in time we're looking at during a flight and, in fact, comes from the same distribution over time.) This last assumption may not hold if, e.g., the error in speed measurements is a function of the plane's speed and the planes traveled at different speeds, or at different speeds at different times of their flights; in practice, this is probably a second-order consideration, but if you're making an important decision, it's probably something to consider.

The approach I would suggest is to calculate the error in the measurements between the two computers for each parameter and then take the mean of this error (again, for each parameter.) This will give 100 samples of the mean error between the two computers for each parameter (one sample from each plane). One can then perform a two-sided t-test for each parameter at a chosen significance level, with the null hypothesis that the mean error between the computers is zero. So, for a given parameter $p$ with $n_p$ measurements $X_{i, p, c, a}$ ($i$ indexing measurements, $c$ indexing computers, $a$ indexing planes), the mean error between the computers is $$\epsilon_{p, a} = \frac{1}{n_p} \sum_i ( X_{i, p, 2, a} - X_{i, p, 1, a} ).$$ To perform a one-sample t-test, calculate $$t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}},$$ where $\mu_0 = 0$ because we're testing the hypothesis that the population mean of the error between the two computers is zero, $\bar{x}$ is the sample mean of the error across the 100 planes, $s$ is the sample standard deviation of the error across the 100 planes, and $n = 100$. If this value of $t$ were outside the acceptance region for your chosen significance level and degrees of freedom, you would reject the null hypothesis and declare that the error between the two computers was statistically significant greater than zero. For example, for a significance level of $\alpha = .05$ and $\nu = 99$ (degrees of freedom $\nu = n - 1$, where $n = 100$ airplanes), R tells me that the acceptance region is $(-1.98 < t < 1.98)$. A significance level of $\alpha = .05$ means that there is a 5% chance of a Type I error, meaning, if this experiment were repeated an infinite number of times, the null hypothesis would be incorrectly rejected 5% of the time.

To come up with one, final decision on whether or not the computers are in good agreement, a crude approach would be to look at what proportion of the hypothesis tests rejected the null hypothesis, and if that proportion were "meaningfully" larger than the chosen significance level, then one would conclude that the computers are in disagreement at a higher rate than mere chance alone would suggest. A better approach would be to use existing theory on combining the results of multiple hypothesis tests, but that's beyond my skill level.

$\endgroup$
0
$\begingroup$

Thank you for this detailed and well written explanation matmat. I can tell a lot of care and passion was involved in your response, and I was prepared to use that answer.

I ended up finding that my delta distributions were not normal and outliers were present, I should have checked that first, and wasn’t able to use the t-test. I ended up needing to use a brute force Monte Carlo bootstrap simulation with replacement because I needed to test sensitivity on number of miles per plane as well as number of planes.

Once the data was simulated with known testing outcomes I used the non-parametric Wilcoxon test to reject or accept the null. Because the data is simulated I know if the null should have been rejected or accepted and could calculate my percent of type 1 and type 2 errors. Finally I determined the acceptable sample size range by doing a sensitivity analysis based on acceptable type 1 and type 2 errors as well as varying scenarios on number of planes and miles per plane.

Thanks so much

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.