I am new to the area of statistics and I am hoping you can suggest methods I may use. Sorry if this is long but I might as well be as clear as possible on my first post :)
What I am worried most is that I may miss out on assumptions and draw conclusions based on statistical tests that, in fact, cannot be applied to my situation.
In a nutshell:
We are replacing a measurement tool + methodology with another tool and a similar methodology and I would like to prove that the new tool & methodology provide the same "results".
The data reported :
Each tool reports 1) the GPS position, 2) a category of measurement (type 1, type 2, type 3) (the categories are the same for both measurement tools and relate to what is being measured, they should report the same thing), and 3) a quantized value of a continuous value. The measurement tools probably quantize the value with different algorithms but, according to spec, they should provide the same value.
Given what we're measuring the measurements are definately not stationnary and, since we're measuring a physical quantity, I assume the time series are autocorrelated.
How the setups differ:
Setup 1 (historical setup) : uses tool "A", takes a measurement 3 times a minute and reports the GPS position, the category of the measurement and the discrete value
Setup 2 (new setup) : uses tool "B", takes a measurement up to every second (but not necessarily based on distance criteria between measurements) and report the GPS position, the category and discrete value too
We put both tools in a car and traveled enough to gather over 100.000 data points for setup 1.
What I would like to prove:
- the categories reported by setup 1 and 2 do not significantly differ
- the discrete value measurements do not significantly differ either
- if the new setup and a bias or skew compared to the other one
What I have done so far:
I have matched each data point of setup 1 to a single data point in setup 2 (the one that is "closest geographically" in a 4 minute-time window). Is this even statistically sound ?
1) Regarding the discrete value reported,
- I drew a scatter plot of the discrete values for matched data points with bubble sizes corresponding to the count for each (x,y) : the data clusters along a 45° angle line as expected but I can see there is some bias.
There is also some spread a round that line
- I drew a Bland-Altman/Tukey diagram of the same data and I now see that the average difference depends on the average mean. That's interesting to know
- I computed the pearson correlation for matches that are in the same category : I get 0.87 which seems to be high enough to look good.
Can Pearson be applied given I have no idea if the distribution is normal and since the measurements are definalty not independent inside the time series ? Would the U test be better ?
- I tried to compute a t test but I'm getting t values in the "80" range because SQRT(N) is huge
I would like to use all the data collected in setup 2 rather than only the data that was matched 1 to 1. There is about 4 times more data reported by setup 2 than setup 1.
I've been looking into non-parametric tests and I believe that is what applies to my case as well as the whole notion of inter-rater agreement. So it seems like my next steps will be to use R to compute Cohen's Kappa and KrippenDorff's alpha.
Would computing these and finding high correlations be enough to make my point ?
2) Regarding categories reported, again the data reported in the time series are correlated because if category 1 is reported then the chance of the next category being reported being 1 is higher than if category 2 had been reported.
Given that there are three categories, what kind of tests could I apply ?
thanks for your suggestions