# Which test do you suggest to use to catch the differences in paired sample

I must evaluate the goodness of an instrument according to the measures of the final products. I have two instruments: one good and the other one whose goodness I have to evaluate. The measures are related to pressures.

What I thought is to sample same products with both the good instrument and the unknown one. The same product will be measured first with the unknown instrument and then with the reliable one. The instruments do not affect each other's measures; they are independent. What I will have is a paired sample.

Without any other information I assume to take about 35 measurements so that I can think of having normality (30 is not a magic number for data normality but the sample size should be defined by the power of the test adopted and I am not sure which test to adopt). Then I will take the difference between the two measurements obtained for each product (to purify the data from the variability of the individual products) in order to obtain a vector of differences which expresses the variability of the two instruments.

What I would like to know is whether the measures of the unknown instrument are significantly different from the measures of the good instrument, I thought about comparing the variance of the sample differences with an ideal population variance (to understand if there is such variability in the data that it can be attributed to different instruments or only to the case). Does this line of reasoning make sense? In this case, can I use F-Test? Which test do you recommend?

• Consider test that involve pairs and allows for correlation, the paired t test if normality is assumed or the Wilcoxon signed rank test for a nonparametric alternative, – Michael Chernick Oct 29 '17 at 20:34
• Hi, thanks for your reply. The paired t test is usually used to determine whether the mean of the differences between two paired samples differs from 0, so is the idea of using variance not valid? – filippo Oct 29 '17 at 21:19
• Power of the test will depend on variance but usually you don't know or have an estimate of it prior to testing. You can do one-sided tests that one mean is greater than the other by a fixed positive quantity. The test will have less power. – Michael Chernick Oct 29 '17 at 23:33

If you wanted, you could test the differences of the unknown from the gold standard with a one-sample t-test. That would be a test of whether the unknown instrument is biased. That would be a test, and people often assume the point of a study is to test something, but it isn't clear that it's the piece of information that should be most important to you. Under the assumption that the vertical spread of the differences was the same across the plot, presumably what you want to know is the standard deviation of the differences, so that you would know how reliable the unknown instrument is, relative to the gold standard. You could also form an $R^2$ type measure by dividing the variance of the gold standard measurements by the sum of the variances of the differences and the gold standard measurements. That would give you an estimate of the proportion of the total variability in the unknown instrument's output that is due to true variability among the products.