2
$\begingroup$

I am currently working on the analysis of a metric comparing two methods (A & B) for subjective annotation. The two methods give 2 types of data which after processing gives scores relatives to the subjective dimension studied for a dataset of item.

In order to compare their performance, I do a reliability test called Split Half Reliability(SHR) that splits the dataset of each method's results in two subsets and compute a correlation coefficient between them. The highest correlation coefficient between two subsets means high reliability. However, there is a lot of variance depending on the organization of the data in each subsets, doing one SHR is therefore not accurate.

So I want to do a bootstrap procedure for each of the method with these steps:

For the two methods'results simultaneously : 1/ sample with replacement the participants data 2/ compute scores for the whole dataset 3/ compute SHR (correlation coefficient, for each method's dataset) 4/ redo steps 1-3 N times

At the end, I'm thinking of a T-test if the distributions of the two methods are normal.

I'm just not sure I'm allowed to follow these steps...

I would gladly take advice (and references if possible).

Thank you :)

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I don't think the split half reliability is the best approach here. The problem with SHR is, as you say, that you can get a different value according to how you split.

Coefficient (Cronbach's) alpha is the average of all the possible split half correlations (it's also a form of ICC). You can calculate the standard error and do a t-test, or you can use a bootstrap, to compare them.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for your answer ! The thing with cronbach's alpha (or krippendorff's alpha) here is that I don't have repeated trials in my dataset. It is due to the design of the method i'm using - i.e. best-worst scaling for many items - and the comparison I'm doing with a rating scale method. So the type of raw data is very different for the two methods, and i have no repeated trials in the whole dataset of best-worst scaling. So the only common metric for the two methods is SHR... $\endgroup$
    – Heralm
    Jul 21, 2021 at 22:33
  • $\begingroup$ Not sure I understand what your data look like. How are you splitting the data? $\endgroup$
    – Jeremy Miles
    Jul 22, 2021 at 16:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.