1
$\begingroup$

(EDITED)

I am curious what methods are out there to model item changes in survey intended to measure the same underlying construct.

Suppose items I1 ~ I4 are used to measure underlying construct C1. Instead of I1 ~ I4, in the following experiment (with different subjects), we use I2 ~ I5 (remove item I1 and add item I5).

  • Experiment 1 : Measure C1 via I1~I4
  • Experiment 2 : Measure V1 vis I2~I5 (removed I1 from experiment 1 and added I5)

  1. What methods can be used to compare C1 observed from I1~I4 vs C1 observed from I2~I5? Can any modeling be done given that response subject is different? For instance, if I would like to know how Subject A who took the Experiment 1 will score on Experiment 2, how can I achieve so?

  2. And also, what methods can be employed if above two surveys were taken in randomized experiment settings? (That half respondents take I1~I4 and the other half take I2~I5 in randomized fashion). For instance, if Respondent A took the Experiment 1 by randomized trial, and I would like to know how he / she will score in Experiment 2, given that we have randomly assigned respondents to A and B, how can I do so?

Thank you!

$\endgroup$
  • $\begingroup$ Sorry, can you add more context? Is this a statistical question or a psychology or something else? (Sounds like IRT) $\endgroup$ – Jon Sep 9 '16 at 18:58
  • $\begingroup$ @Jon I have updated some details; how does IRT work in this context? I do have basic understanding of 1PL / 2PL IRT, but have no clue on how it relates to these item changes' effect on construct measurement. $\endgroup$ – won782 Sep 9 '16 at 19:28
2
+50
$\begingroup$

This likely won't be the most complete answer, but it should point you in the right directions and clarify a few limitations in what you want to do.

What methods can be used to compare C1 observed from I1~I4 vs C1 observed from I2~I5? Can any modeling be done given that response subject is different? For instance, if I would like to know how Subject A who took the Experiment 1 will score on Experiment 2, how can I achieve so?

I'll focus on the specific instance that you mention: You have a person's responses on I1~I4, and you want to predict their score on I2~I5.

The possibilities here are limited for a reason that should hopefully become obvious. While you already have some data covering their responses to items I2, I3 and I4, you know nothing about how they might respond to I5. Within item response theory a key idea is that a set of items measuring a single construct exist on a hierarchy of difficulty (IRT terminology is influenced by educational testing, and "harder" items are those that require more of the construct to give an affirmative response). This means that there are some items that are more likely (IRT models are stochastic) to receive higher responses than others.

Using, for example, a Rasch model (a 1-parameter IRT model) you can place each item in order of difficulty, but of course you can only do that as long as you have data about people's responses. You have no data about I5. You don't know if it's easy, hard, or in the middle. So while you could use the Rasch model to predict peoples' responses to I2, I3, and I4, you couldn't estimate the probability of them giving a specific response to I5 (IRT is often about estimating the probability of individual items, rather than total scores).

Now, this becomes possible if you do have some data about I5...

And also, what methods can be employed if above two surveys were taken in randomized experiment settings? (That half respondents take I1~I4 and the other half take I2~I5 in randomized fashion). For instance, if Respondent A took the Experiment 1 by randomized trial, and I would like to know how he / she will score in Experiment 2, given that we have randomly assigned respondents to A and B, how can I do so?

If you have one group of people responding to I1~I4 (group A) and another group responding to I2~I5 (group B), and people have been randomly assigned to groups, then, as long as the merged data fits the Rasch model, you can predict how a person in group A might respond to I2~I5. This is because the Rasch model permits invariance. Roughly speaking, this means you can separate estimates of the difficulty of the items from estimates of the ability of the people. Because you have a measurement of a person from group A's ability (which comes from their responses to I1~I4) and a measurement of I5's difficulty (which comes from group B's responses to I2~I5), you can then estimate the probability of them giving a certain response to I5 despite the fact you have no data about their responses to I5.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.