3
$\begingroup$

I am trying to understand a research result. Imagine 10 different psychological questionnaires 1,2,3,...10 of different length (i.e. number of questions) that are all supposedly measuring the same construct. Each scale has between 10 and 30 questions.

I now compute the Jaccard Index J ranging from 0 to 1 for each pair of scales:

J = a/(a + b + c), where

a = number of items shared by 2 scales,

b = number of items unique to the first scale,

c = number of items unique to the second scale.

Afterwards, I compute the average Jaccard Index for each scale (with all other scales).

Curiously, in my empirical example, this averaged coefficient for each scale correlates 0.5 with the number of items per scale — the longer the scale, the higher the Jaccard Index.

I want to understand if this is due to chance, or some property of the Jaccard index, and would very much appreciate feedback.

$\endgroup$
0
$\begingroup$

I'm confused when you said "the average coefficient of each scale correlates 0.5 with the number of items per scale". Isn't there a coefficient for each pair of scales?

However, my intuition is that you get more interpretable results when comparing sets of similar size. Also, that small sets (anectodally I think smaller than 5) can become an issue. I concluded this when I was dealing with a problem where I started off comparing sets ranging from 1 - 200+ items. My logic was as follows:

Set A is small, Set B is large = can never get a large J-index

Set A is small, Set B is small = few items overlapping can give a large J-index

Set A is large, Set B is large = need many items overlapping to get a large J-index

Essentially, since the distribution of possible Jaccard indices is dependent on set size (i.e. two sets of size 2 can only give 0, 0.33, or 1); then the shape (mean, variance, etc.) of the resulting distribution will also depend on the mean/variance of the overlap between the sets your comparing. If you observe larger sets have greater overlap, I would be concerned there may be a hidden covariate underlying the design of your scales. Maybe the respondents were affected by the length of the scale they were given. Or were the scales constructed by two different people, but the randomization led to one person constructing most of the long scales (and vice versa)? I simply don't have enough information about what types of questions there may be, how they were split up within/between scales, and how the data was collected, to form a strong opinion about your results.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.