# Determining the right method to compare observed versus measured data

I'm trying to solve this problem out loud. I'm not great at statistics but I try sometimes to understand where things are and where they're going.

I'm trying to figure out the best method to compare the two following datasets. The first is the actual observation and the second is of what my sensor determined the value was. Basically an object travels across the sensor that with properties. Each object can have anywhere from 2 to 5 components. We know the exact value of each one of those components.

The big table below shows the data results from running those objects on the sensor. Observations are independent so observation 5 has no affect on observation 26. The dataset has 31 objects in it, and each object was measure using the sensor. We know the exact properties of the object and I put it under "Actual"

Note:

Total = 1 + 2 + 3 + 4 + 5

All what I'm trying to figure out is if the sensor was adequate or does it need further calibration? The old way of doing things was to get an average difference less than 5% to be acceptable. I don't agree with that methodology as I believe it's prone to error and does not capture the essence of measurements. So therefore, I'm investigating better data analysis tools for the problem.

Hypotheses:

1. "Is the sensor performing adequately?"
2. Is there an overall statistical difference between Actual and Sensor values?
3. Is there a statistical difference between each observed pair?
4. At which point is the significance broken? i.e. at which confidence interval do the two datasets become different?

Analysis:

The first thing I did was to compare the average differences for each column. I also did the mean of the absolute differences and here are the results:

   Difference between Actual and Sensor (%)
Total        1        2      3         4        5
Mean of Diff        1.06     2.80    -4.00   7.12    -10.86    17.01
Mean of abs diff    7.31     6.96     7.56   8.52      0.70     1.10


These numbers don't tell me much really but they show somewhat of a relevant comparison. I then proceeded to do a two paired t-test using alpha = 0.05 with the following results:

        df      Mean Actual     Mean Sensor     Var Actual      Var Sensor      pearson     t stat      P 1 tail    t Crit 1 tail   P 2 tail    t Crit 2 tail
Total   30      37,858          89,829,589      89,829,589      77,766,451      0.9463      1.4871      0.0737      1.6973          0.1474      2.0423
1       30      5,411.61        5,261           254,000         407,118         0.7829      2.1076      0.0218      1.6973          0.0435      2.0423
2       30      15,246          15,719          4,762,169       3,878,279       0.7845      -1.9103     0.0328      1.6973          0.0657      2.0423
3       27      17,943          15,396          40,196,438      33,216,653      0.4268      2.0742      0.0239      1.7033          0.0477      2.0518
4       1       6,770.00        7,600           14,257,800      20,480,000      1.0000      -1.5661     0.1809      6.3138          0.3618      12.7062
5       1       5,910.00        4,900           64,800          0.00            N/A         5.6108      0.0561      6.3138          0.1123      12.7062


I basically concluded that the means are not significantly different for "Total", "", "4", and "5". Am I correct on the analysis?

I then did an F test analysis for two means to compare those results. Note that I wasn't sure if I needed to do the F-test but I did it to compare the variance. I concluded that according to the F test, it appears that my variance is high on all accounts that the results are statistically significant.

        F   P 1 tail    F Crit 1 tail
Total   1.1551  0.3477      1.8409
1       0.6239  0.1011      0.5432
2       1.2279  0.2887      1.8409
3       1.0299  0.4698      1.9048
4       0.6962  0.4427      0.0062
5       65,535  N/A         161.4476


Data Table

            Actual                                             Sensor
Observation   Total       1       2       3       4       5    Total      1       2       3       4       5
1   36820    5860   14840   16120                   36200    5500   15900   14800
2   42530    5160   12000    9840    9440    6090   43300    5200   13100    9200   10800    4900
3   48650    4770   18260   25620                   52000    4700   18300   29000
4   42290    5020   16080   21190                   48800    5600   18400   24800
5   40530    5350   15670   19510                   37800    4800   15300   17700
6   41610    5140   16000   20470                   41500    5700   17100   18700
7   35210    5310   15150   14750                   34000    5100   15600   13300
8   36150    5430   15520   15200                   33600    4900   15400   13300
9   17600    6770   10830                           19300    6600   12800
10   35920    5280   15540   15100                   28700    4700   13000   11000
11   37900    5740   15440   16720                   35100    5500   14700   14900
12   41530    5590   16560   19380                   39300    5400   16600   17200
13   33580    5330   14870   13380                   33400    5100   14800   13500
14   25550    4990   13450    7110                   24600    4800   13000    6800
15   45920    5420   16200   24300                   43300    5100   15800   22400
16   17750    5740   12010                           21700    6500   15200
17   29570    5230   14140   10200                   28800    4600   14600    9600
18   44870    5150   16990   22730                   47500    5600   18900   23000
19   35500    5780   14930   14790                   32200    5600   14200   12500
20   54070    5310   17170   31590                   50100    5000   16700   28300
21   35440    4590   13920   16930                   35000    3900   14400   16600
22   40680    5330   12840   12680    4100    5730   40000    5400   13400   11900    4400    4900
23   16540    7160    9380                           20200    6900   13300
24   47820    4960   17480   25380                   47100    4700   18100   24300
25   39410    5350   18070   15990                   42100    5900   20700   15500
26   40790    5470   16480   18840                   36400    5000   15700   15700
27   31760    5480   14570   11710                   33700    5900   16200   11600
28   33000    5090   14760   13150                   30900    4800   14600   11500
29   49390    5350   18110   25930                   45500    4900   18100   22400
30   54290    5110   17540   31640                   48800    4800   16700    27300   20900
31   40930    5500   17850   17580                   37300    4900   16700   15700

• Just to clean up the question a little: it appears you have reversed the interpretation of "significant": small p-values indicate significance while larger ones do not. Also, people will be wondering whether there is a natural ordering to the observations (such as being sequential in time) or whether they should be considered independent of one another. – whuber Feb 8 '12 at 20:44
• Thanks for the p-note significance, I always get confused :) hopefully now it stays with me for ever. On your other note, the observations are independent in terms of observation 5 has no affect on observation 26. – dassouki Feb 8 '12 at 20:50
• You should consider editing the question to reflect this. For example, take another look at the t-test results. I'm sure people will also soon be pointing out the need to adjust for multiple comparisons, so you might want to get the jump on that by searching this site on multiple-comparisons or "Bonferroni". Your data are interesting in that this adjustment makes a substantial difference: it's not merely of theoretical concern. – whuber Feb 8 '12 at 20:54
• @whuber I'll try and do that tonight or tomorrow. Right now I'm reading about Bonferroni – dassouki Feb 8 '12 at 21:00
• I assume "1" is the same component in both series, etc. Have you tried plotting Actual 1 versus Sensor 1, 2v2, 3v3 as separate scatterplots to eyeball the data? – Michelle Feb 20 '12 at 9:10

There are a number of things you can do with this sort of data but in the end you have to make some calls about what is acceptable to you - and the old way of saying "is the average error more than five percent?" may not be such a bad way of doing it.

Statistics can be useful in showing whether there is bias in the measurements; and in whether any of the properties is systematically mismeasured. And it can help quantify on average how far out the sensor is, and a sense of the spread of that (ie is it mostly right and occasionally way out - or always mediocre). It's your judgement call on whether that's acceptable.

Here's what I did with your data (ok, I was having a slow afternoon...), using R and Hadley Wickham's plyr and ggplot2 libraries.. My first instinct (basically also Michelle's suggestion) was to look at the relationship between the actual value and the sensor's value, conditioned on the five properties. That gives me:

which is a start. Amongst other things I see you have very few observations on properties four and five, which might complicate things later. And that some of the properties typically have much higher scores than others.

Next, I built a similar plot and compared the data to a line with zero intercept and slope of 1 which is what you'd get if the sensor and the actuals were always the same. I added to this locally smoothed lines showing the actual relationship between actual and sensor scores for the first three properties. This gives me the following, which is starting to suggest that property two is maybe scored higher than it should be by the sensor; and properties one and three perhaps the opposite problem. Property three is probably the one most to look out for, judging by this plot.

I'm interested in testing this in a model, but there is clear heteroscedasticity ie the variance in sensor scores, unsurprisingly, increases as the scores increase. This would invalidate the simpler models to fit, so I try taking logarithms and this problem goes away:

I can then use this as the basis of fitting a linear regression model which finds that yes, there is statistically significant evidence that knowing which property is being measured is helpful in knowing what the sensor's score will be. If the sensor was equally good at measuring all properties you wouldn't get this, so we now know there is a problem here at least (again, whether it matters is up to you).

> mod <- lm(log(sens) ~ log(act) + act.var, data=sensorm)
> anova(mod) # shows there is a difference in how well properties measured
Analysis of Variance Table

Response: log(sens)
Df Sum Sq Mean Sq F value  Pr(>F)
log(act)   1  30.15   30.15 3854.54 < 2e-16 ***
act.var    4   0.28    0.07    9.06 3.6e-06 ***
Residuals 88   0.69    0.01
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> summary(mod) # as graphics show, property 2 seems higher than properties 1 and 3
...
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)  0.37988    0.35210    1.08    0.284
log(act)     0.95216    0.04094   23.26   <2e-16 ***
act.varX2    0.11459    0.04772    2.40    0.018 *
act.varX3    0.00855    0.05262    0.16    0.871
act.varX4    0.14067    0.06479    2.17    0.033 *
act.varX5   -0.15136    0.06463   -2.34    0.021 *
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.0884 on 88 degrees of freedom
(61 observations deleted due to missingness)
Multiple R-squared: 0.978,      Adjusted R-squared: 0.977
F-statistic:  778 on 5 and 88 DF,  p-value: <2e-16


Ok, so I've resolved that at least one of the properties is measured differently from the others. Next I wanted to get a feel for how far out in general the measurements are. I did this by plotting a histogram of the average percentage out for each observation of a property - which shows there actually is a pretty big range:

I'd intuitively say that already this is enough to indicate you need some more callibration, but we can quantify it a bit.

There's not quite evidence the average is significantly different from zero ie of systematic bias, but it's getting close (p value of around 0.08). If it weren't for those couple of big values where the sensor was 20-40 percent too large, you'd say that it seems to generally underestimate the true value.

However, perhaps more importantly, the average absolute value of the percentage out is about seven or eight percent (depending on how you interpret "average") which strikes me as too much. But see, we've just come back to that first rule of thumb, albeit after some useful graphical insight...

Hope that helps. The code in R that produced this is below.

sensor <- read.csv("sensor.csv", row.names=1) # reads data in as matrix with 12 columns

library(plyr)
library(ggplot2)

sensorm <- cbind(melt(sensor[,2:6]), melt(sensor[,8:12]))
names(sensorm) <- c("act.var", "act", "sens.var", "sens")
win.graph()
qplot(act, sens, data=sensorm, facets=.~act.var)

win.graph()
qplot(act,sens, data=sensorm, colour=act.var) +
geom_abline(intercept=0,slope=1, legend=F) +
geom_smooth(data=sensorm[as.numeric(sensorm\$act.var)<4,],
legend=F, se=F) # only draw this line for 3 properties,
# as properties 4 and 5 have too few points

last_plot() + scale_x_log10() + scale_y_log10() # seems to fix heterosceadsticity ie variance
# now roughly the same at different values

mod <- lm(log(sens) ~ log(act) + act.var, data=sensorm)
anova(mod) # shows there is a difference in how well properties measured
summary(mod) # as graphics show, property 2 seems higher than properties 1 and 3

qplot((sens-act)/act*100, data=sensorm)
with(sensorm, t.test((sens-act)/act*100)) # not quite evidence of bias, but nearly significant
with(sensorm, mean(abs(sens-act)/act*100 , na.rm=T)) # on average 8% wrong
with(sensorm, mean(abs(sens-act)/act*100 , trim=0.2, na.rm=T)) # on average 7% wrong even when trimmed

with(sensorm, cbind(act, sens, round(abs(sens-act)/act*100))) # gives table of actual, sensor, and % out

• I think this is the exact answer I was looking for. I will be reading through it a few more times on Monday. – dassouki Feb 25 '12 at 15:44