From what I understand, the paired t-test is used when measuring the same "thing" in two different scenarios (e.g. one student's pre-test vs. post-test scores). However, is there a test that takes into consideration the variation in the measurment? Take the data set below. The objective is to determine if the two instruments produce similar results (Data 1 vs. Data 2). The testing is destructive so a direct paired t-test is not possible. However, each batch of material (15 in total) is tested in triplicate on each instrument. There is batch to batch variation, so I feel like doing a non-paired t-test loses that information. What is the best way to determine if the instruments are statistically indistinguishable?
-
2$\begingroup$ A multilevel, or mixed, model is designed to solve this problem. $\endgroup$– Jeremy MilesCommented Aug 8, 2018 at 19:37
-
1$\begingroup$ Thanks! I am not familiar with a this model. Can you point me to where I can learn about it? $\endgroup$– KoldBeansCommented Aug 8, 2018 at 20:25
-
1$\begingroup$ I think I got it. I assume it is a multilevel regression, and if the coefficient for "instrument" is statistically significant, then the instrument is a factor that affects the measurement? $\endgroup$– KoldBeansCommented Aug 8, 2018 at 20:29
-
$\begingroup$ Repeated measures ANOVA, or if normality and sphericity assumptions are not tenable, the Friedman test? $\endgroup$– AlexisCommented Jan 3 at 3:11
1 Answer
In experimental design your batches of material is called blocks. Then the blocks/batches are modeled as a random effect, and the A/B-instrument comparison is done within blocks.
You can solve this using a mixed model. You are measuring something (two different instruments, the treatments) on material that comes in batches. The idea is to divide each batch in two (roughly equal) parts, and using instrument A on one part, instrument B on the other.
A model can be written as $$ Y_{ijt} = \mu + b_i + \beta_t + \epsilon_{ijt} $$ where $i$ is batches, $j$ is material within batch, and $t$ (A or B) is instrument. $b_i$ is a random batch effect, $\beta_t$ is treatment effect. In R this can be done as (not tested)
library(lme4)
mod <- lmer(Y ~ instrument + (1 | batch), <etc>)
This model is assuming constant variance of the error term $\epsilon_{ijt}$. This is a variance component model, see What is a variance component model?