How to "prove" that new measurement tool & process gives same result as old?

Question

I am new to the area of statistics and I am hoping you can suggest methods I may use. Sorry if this is long but I might as well be as clear as possible on my first post :)
What I am worried most is that I may miss out on assumptions and draw conclusions based on statistical tests that, in fact, cannot be applied to my situation.

In a nutshell:
We are replacing a measurement tool + methodology with another tool and a similar methodology and I would like to prove that the new tool & methodology provide the same "results".

The data reported : Each tool reports 1) the GPS position, 2) a category of measurement (type 1, type 2, type 3) (the categories are the same for both measurement tools and relate to what is being measured, they should report the same thing), and 3) a quantized value of a continuous value. The measurement tools probably quantize the value with different algorithms but, according to spec, they should provide the same value.
Given what we're measuring the measurements are definately not stationnary and, since we're measuring a physical quantity, I assume the time series are autocorrelated.

How the setups differ:
Setup 1 (historical setup) : uses tool "A", takes a measurement 3 times a minute and reports the GPS position, the category of the measurement and the discrete value
Setup 2 (new setup) : uses tool "B", takes a measurement up to every second (but not necessarily based on distance criteria between measurements) and report the GPS position, the category and discrete value too

Our experiment:
We put both tools in a car and traveled enough to gather over 100.000 data points for setup 1.

What I would like to prove:

• the categories reported by setup 1 and 2 do not significantly differ
• the discrete value measurements do not significantly differ either
• if the new setup and a bias or skew compared to the other one

What I have done so far:
I have matched each data point of setup 1 to a single data point in setup 2 (the one that is "closest geographically" in a 4 minute-time window). Is this even statistically sound ?

1) Regarding the discrete value reported,

• I drew a scatter plot of the discrete values for matched data points with bubble sizes corresponding to the count for each (x,y) : the data clusters along a 45° angle line as expected but I can see there is some bias.
  There is also some spread a round that line
• I drew a Bland-Altman/Tukey diagram of the same data and I now see that the average difference depends on the average mean. That's interesting to know
• I computed the pearson correlation for matches that are in the same category : I get 0.87 which seems to be high enough to look good.
  Can Pearson be applied given I have no idea if the distribution is normal and since the measurements are definalty not independent inside the time series ? Would the U test be better ?
• I tried to compute a t test but I'm getting t values in the "80" range because SQRT(N) is huge

I would like to use all the data collected in setup 2 rather than only the data that was matched 1 to 1. There is about 4 times more data reported by setup 2 than setup 1.
I've been looking into non-parametric tests and I believe that is what applies to my case as well as the whole notion of inter-rater agreement. So it seems like my next steps will be to use R to compute Cohen's Kappa and KrippenDorff's alpha.
Would computing these and finding high correlations be enough to make my point ?

2) Regarding categories reported, again the data reported in the time series are correlated because if category 1 is reported then the chance of the next category being reported being 1 is higher than if category 2 had been reported.
Given that there are three categories, what kind of tests could I apply ?

thanks for your suggestions

Answer 1

First of all, a question if interest: If you want the measurements not to differ significantly, why change the tool at all? Simply to get more frequent measurements, or for economical reasons?

Now for the reply.

I do not entirely understand how you gathered the data. You say both instruments were colocated, and that instrument B gathered data more frequently. For comparison purposes, you would want those measurements of B that were made at the same place as A. As you cannot match them exactly, you need to interpolate. For this, I would use timestamps, but assume you don't have them, as you went with the GPS coordinates (although you did say "4-minute interval"). I'll assume it's reasonable to assume the GPS coordinates are accurate for both instruments, and that the effect measured doesn't noticeably vary over very small distances, such as the location the instruments are in on the vehicle. For interpolation, you need a model of the variability of the measured effect between datapoints. You say you used the nearest neighbour. That's perfectly acceptable, but I would probably go with a linear interpolant myself. Using all datapoints would require a very good model, which you may not yet have.

As for the comparisons:

• You found that the instruments do differ, depending on the true value. This means at least one of the instruments is biased, but you will not be able to tell from the measurements alone which one. You may be able to fit a generalized linear model

• The correlation you measured is actually not all that good for what is supposed to be 1. The Pearson correlation does not require normality, but it does assume the variables are stationary and independent - which yours aren't.

• Large N isn't a problem, it's a good thing. Your large t-value also tells you that if your variable is normally distributed, the instruments are not equivalent.

• In testing the reported categories, start with a simple crosstabulation to see if any particular mismatch is more likely than others. That the time series is autocorrelated doesn't matter when looking just at the differences. To test for independence of the categories, you first need to determine whether categories depend on the discretized variable, and if not, whether they nevertheless are ordered or merely unordered labels. The choice of test comes after you characterize how you think the data should behave.

Answer 2

Another question seems to have covered similar ground, at least in regards to similarity of discrete values. There I suggest regressing regression would be usable in theory for the GPS position, though I imagine a solution that respects the two dimensionality of the data would be preferable. @chi has a better answer on that same question (that includes citations). Similarity of distributions seems to be a question for a K-S test (I wonder if there is a multivariate version). For the categorization data it seems like Baaysean approach could be useful. Given categorization A at time slice 1 using Method 1... what is the Pr(Cat B) @ time slice 1 using Method 1? You might also do a lagged version of this to look at whether one method is picking up categorization changes before or after the other.