# Two test groups with multiple measurements vs a single reference value

This is my scenario:

I'm testing two length measuring devices. I'm measuring a model that has notches at different lengths in order to collect 15 different measurements. I know the "real" value for each distance in order to calculate 15 "errors" for each device. The same 15 measurements are repeated ten times for each device.

• Device A - 15 measurements x 10 times
• Device B - 15 measurements x 10 times

In order to have a general idea about which one is better I thought that a t-test would be ok (tell me if not): I put all the errors of Device A together and compare them with B.

If I want to compare A vs B of each one of the 15 measurements would it be ok to do a one way ANOVA? I'm asking it because I have only two groups.

• Do the real values vary? (i.e. are they always measuring 15cm, or is it sometimes 10cm, sometimes 20cm, etc.) If that's the case then an alternative approach may be to calculate correlation coefficients for each device-real pairing, and look to see which has the larger coefficient. Sep 15 '16 at 10:01
• I'm not sure I understood correctly. I have 15 "known" distances, eg. 13 mm, 14, 18, 18,6, etc... And I want to know which one is closer to the real distances
– Fed
Sep 19 '16 at 12:33

If you just want to compare the differences between the two groups than a hypothesis test like a t-test or a Wilcoxon test is the most convenient way. There are some differences between statistical tests regarding small sample properties and how they deal with different variances. For reasons of simplicity I propose a simple t-test (welche two sample t-test).

Let´s have a look a two vectors. The first vector is called "a".

2 4 3 5 6 4 2 7 8 4


The second vector is called "b".

6 3 4 2 6 8 8 6 8 4


So you can use the following R command for testing.

t.test(a,b)

Welch Two Sample t-test

data:  a and b
t = -1.0674, df = 17.897, p-value = 0.3
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-2.9691637  0.9691637
sample estimates:
mean of x mean of y
4.5       5.5


The null hypothesis is that both samples have the same mean. The alternative hypothesis is that there are significant differences between the values of the two vectors.

One-way ANOVA however is applicable if you want to compare means of three or more samples. As you have only two samples you should not use a one-way ANOVA.

• There are two issues with this approach. Firstly, depending on how the errors are summed the mean could likely be zero for both groups despite the devices varying wildly in their accuracy. Secondly, this assumes that both devices measure on the same scale. If the scales are different then two similarly (in)accurate devices could have different mean errors. Sep 15 '16 at 12:39
• Thank you very much for your comment. Regarding the first issue: Of course one should have two compute the sum of absolute errors or the sum of squared errors. Regarding the second issue it would be presumably sufficient to transform one of the two vectors by dividing them or by transforming them using z-values, inverse hyperbolic sine or logarithmic transformation. Sep 15 '16 at 12:46
• @Ferdi Thanks a lot For the answers. I applied the t-test for the "overall" comparison between the two machines. Nevertheless, what if I would like to perform statistics for each measure?
– Fed
Sep 19 '16 at 12:37
• @Ferdi Thanks a lot For the answers. I applied the t-test for the "overall" comparison between the two machines. Nevertheless, what if I would like to perform statistics for each measure? here is a diagram of the measurements made [link] (s22.postimg.org/wuecmndch/frecce_Misuraz_001.jpg) Here is a sample of the 15 measurements made with Device A, repeated ten times. There is a similar table for Device B link
– Fed
Sep 19 '16 at 12:42

Following extensive discussion in the comments with the OP, this approach is likely inappropriate in this specific case, but I'll keep it here as it may be of some use in the more general case

With your data you have three different measurements:

First, you have the "reference" measurement, i.e. the thing you are interested in measuring. For simplicity's sake, let us assume that this is known without error.

Second, you have the measurement taken from Device A. This is a measurement of the reference object which has some error.

Third, you have the measurement taken from Device B. Again, this is a measurement of the reference object which has some error (which may be more or less than the error with Device A).

The error associated with both measurement devices ensures that there will be variance in both sets of measurements. Furthermore, as you have a range of reference values (i.e., you didn't just measure the same thing multiple times) you'll have some variance in the reference measurement.

You could calculate a correlation coefficient between the reference measurement and the measurement from each device. The closer the coefficient is to 1 the more the variance in your measurements can be accounted for by the variance in the reference measurement, and therefore the less error there is (error is the variance that you can't account for by knowing the length of the object being measured).

For a specific sample, the device with the largest correlation coefficient (i.e., closest to 1), will be the less errorful device.

An example

As an illustration, I'll set up data for two measurement devices. One which is more errorful than the other

reference <- 6:20
measureA <- reference + rnorm(15, 0, .5)
measureB <- reference + rnorm(15, 0, 3)


And now, lets compare the measurements for each device with the reference measurements

It should hopefully be clear here that there is more error associated with device B.

Now, we can calculate correlation coefficients for each device compared to the reference

> cor(reference, measureA)
[1] 0.9954188
> cor(reference, measureB)
[1] 0.6487995


Note that the device with more error has a smaller correlation coefficient than the one with less error.

Of course, you may want to know whether the difference between correlation coefficients is statistically significant. This question may give you some help in that direction, although with only 15 observations the differences in reliability between the two devices may need to be large before you get a significant $p$-value.

Note:

For this approach, it won't matter whether the two devices are measuring on the same scale as the correlation coefficient is standardised.

• As a reference measure I have only one value. If I run correlation with SPSS duplicating ten times the reference measure, I get an error because one set of data (reference measure) is constant. If I place all the 15x10 measurements in one column, I can see the overall correlation but not each one of them. Am I missing something?
– Fed
Sep 19 '16 at 16:24
• In your earlier comment you said that you had 15 known distances, which varied. The reference measures are these known distances. Am I misunderstanding something? As I understand it, you essentially have 15 distances which you've measured with each of your measuring devices Sep 19 '16 at 16:33
• Thank you @Ian_Fin for the patience "15 known distances, which varied" --> right. Different segments with known distance (because i measured it with a reference machine). In the experiment, segment #1 to #15 were measured ten times each with both machines. I would like to be able to test significance between device A and B for each one of the segments
– Fed
Sep 21 '16 at 9:06
• @Fed So you have 15 different segments of known, and varying, distances, and for each measurement device you have 15 measurements (one for each segment)? I don't understand where the duplication comes in, unless you measure each segment multiple times with the same device Sep 21 '16 at 9:10
• Yes I do: I repeated the scan of the whole object (that has 15 measurements points within) ten times for each device. For each one of the 15 segments, I have 1 real value, 10 values for device A and 10 values for device B
– Fed
Sep 22 '16 at 6:49