Description of my experiment: I have 2000 samples. For each sample I am measuring a metric Mi using four different methods A,B,C and D. These methods of measurement are completely independent. Ideally, the measurements should result in the same values for the metric i.e; M1A=M1B=M1C=M1D However, in some metrics, the different methods result in different values (subjective opinion based on reading the values for a few samples).

To test whether these values are statistically different I considered MiA as my reference and carried out paired t-tests between MiA and MiB; MiA and MiC; MiA and MiD, with the null hypothesis that the mean difference is 0 (2000 samples)

The results seemed alright to me

I also did a simple linear regression (SLR) with MiA on the X-axis and MiB, MiC and MiD. Visually there was a lot of information that was apparent which I liked. In some cases, the slopes were different from 1 and intercept was not 0. My understanding here is that if there are no differences, the points should fall very close to the y=x line.

I then carried out a joint hypothesis test (linearHypothesis() in R) where slope=1 and intercept!=0. In this test, I was to some extent able to relate the results to the visual effect of the scatter plots (when the lines are very different).

Is there a way to make sense of the results from the t-test and SLR together?

Although they both seem to convey some useful information, I feel limited in my ability to express their meaning together. As it is evident, the number of p-values is a lot! Besides the mean differences(ttest) and slope and intercept values for each model...

Example images for two different metrics are shown below, (MiB~MiA is green, MiC~MiA is orange, MiD~MiA is purple)

enter image description here enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.