1
$\begingroup$

I am conducting a study on a pool of data which consisted of 200 items. I am planning to use rating scales to see how these items relate to a concept (e.g. "how are the next items related to the attractiveness of a product?") and the rating scales ranges from 1-very closely related, to 7-not related at all.

The analysis includes reducing the number of items to about 15-20 that are most relevant and describe the concept of the study most effectively. Now I think that 200 items are way too much for one person to rate, so I was planning to split my pool of data to smaller groups and ask each participant to rate only a group of items (e.g. 25 items). This means that participant 1 to 30 evaluates item 1 to 25, participant 31 to 60 evaluates item 26 to 50 and so on...please note that we assume that all these items are equally related to the concept of the study, so participants will be randomly assigned to the items.

My question is: Could this method potentially comprimise the validity of my results? Do you have any tips for the analysis?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
1
$\begingroup$

Since you stated "The analysis includes reducing the number of items to about 15-20 that are most relevant and describe the concept of the study most effectively", an easy(er) solution to this would be to reduce the number of items to about 15-20 for ALL participants to complete. This way you have nothing to worry about in terms of validity related to item selection later on in the analysis phase! This was a great consideration a priori as it will reduce your workload later on

Improve this answer
$\endgroup$
2
  • $\begingroup$ That is what I'm trying to do, 15-20 items for all participants. but the thing is, a participant does not have access to all the items at once but rates only a fraction of all items and only once. the rest of the items are rated by other participants. so I am concerned that this could introduce some sort of bias to their ratings or comprimise the validity because certain number of items in the pool are rated by different people. I hope it is clear since my knowldge of statistics is poor... $\endgroup$
    – user29166
    Commented Aug 20, 2013 at 19:40
  • $\begingroup$ this WILL introduce inherent bias to your study, which is why I suggest picking the most important 15 to 20 items for ALL of the participants to complete. In this sense, you will have complete data on all participants. This results in much more interpretable results and the need for fewer participants without having to worry about these inherent threats to validity. $\endgroup$
    – Matt Reichenbach
    Commented Aug 20, 2013 at 19:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.