I am currently working on the analysis of a metric comparing two methods (A & B) for subjective annotation. The two methods give 2 types of data which after processing gives scores relatives to the subjective dimension studied for a dataset of item.
In order to compare their performance, I do a reliability test called Split Half Reliability(SHR) that splits the dataset of each method's results in two subsets and compute a correlation coefficient between them. The highest correlation coefficient between two subsets means high reliability. However, there is a lot of variance depending on the organization of the data in each subsets, doing one SHR is therefore not accurate.
So I want to do a bootstrap procedure for each of the method with these steps:
For the two methods'results simultaneously : 1/ sample with replacement the participants data 2/ compute scores for the whole dataset 3/ compute SHR (correlation coefficient, for each method's dataset) 4/ redo steps 1-3 N times
At the end, I'm thinking of a T-test if the distributions of the two methods are normal.
I'm just not sure I'm allowed to follow these steps...
I would gladly take advice (and references if possible).
Thank you :)