0
$\begingroup$

I was wondering whether I could overcome (assumed) measurement invariance by rescaling continuous items.


Example

Let's say I have the following two questions (simplified for illustration):

Imagine student X enters the stage during a graduation ceremony to shake hands with the dean.

How loudly would students in the audience applaud?

How loudly would teachers in the audience applaud?

For both questions I use 100 point sliders with the following two anchors at the respective scale ends: 1: Crossed arms and no clapping - 100: Fierce applause and cheering.

Edit: What I am interested in is the degree to which students/teachers in the audience support student X, not how supported the student feels.


The Problem

I believe that in both cases higher values indicate more support for the student on stage, but also believe that teachers will not be expected to cheer or applaud as enthusiastically as students. In other words, a relatively lower response for the degree to which teachers (vs. students) would applaud might signal the maximum support available from teachers. In contrast, the maximum support available from students might indeed be 100 (or close to 100) on the 100 point scale.

Can I overcome this problem of measurement invariance (that e.g., 75 means maximum support from teachers but that e.g., 95 means maximum support from students) by rescaling both questionnaires so that their minimum and maximum respectively map to 1 and 100 like on the original 100 point scale?


Additional Thoughts

  • I read on several posts (e.g., this one that rescaling might not be a good idea because the original scale had a meaning to participants that is not retained in the rescaled version. However, I believe that exactly this problem (that the meaning of maximum support differs based on whether it is students or teachers who applaud) is an argument for rescaling in this case.

  • Alternatively, I was thinking of standardizing variables and using z-scores, but based on the above logic the standard deviation for applause from students should be larger than the standard deviation for the applause from teachers.

  • I realize I could change the scale anchors, e.g., remove the "cheering" aspect of the scale and thereby limit the response to degrees of applause. Although that might solve the problem that teachers might not be expected to cheer to individual students during a graduation ceremony, I feel I would unnecessarily restrict response options for applause from students, who might very well cheer.

$\endgroup$
8
  • $\begingroup$ Could the fact that a "maximally supported" student might receive 95 on the student scale and 75 on the teacher scale not be seen as something similar to difficulty within IRT, particularly with polytomous scales (i.e. if items used a four-point rating scale then a person may be reasonably expected to respond with four on an "easy" question, but respond with 3 on a more difficult question)? As long as both items are reflecting the same latent construct (the support for the student) then it doesn't seem problematic. $\endgroup$
    – Ian_Fin
    Sep 23, 2016 at 13:51
  • $\begingroup$ @Ian do you mean rescaling does not seem problematic in that case or do you mean that measurement invariance does not seem problematic? I suspect the former but just wanted to ask to be sure. $\endgroup$
    – Flo
    Sep 23, 2016 at 14:18
  • $\begingroup$ Actually, I suspect it's closer to the latter. To an extent though, I think it may depend on what it is you're actually measuring (or at least the purpose of measurement). One of the questions that keeps coming to my head is whether you're interested in how supported the student feels (perceived support) or how the teachers/peers feel about them (actual support). Support being capped for teachers is problematic in the one case, but not the other $\endgroup$
    – Ian_Fin
    Sep 23, 2016 at 14:45
  • $\begingroup$ It also seems to me that the validity of rescaling depends on whether you think the teachers rescale their enthusiasm to reflect the cap, or simply "filter out" the high end enthusiasm. $\endgroup$
    – Ian_Fin
    Sep 23, 2016 at 14:55
  • $\begingroup$ @Ian thanks for bearing with me! I am interested in actual support (how teachers/peers feel about the graduate) rather than felt support by the graduate him/herself. I was thinking that teachers--when they show their fullest support--will show less applause/cheering than students who show their fullest support. So perhaps a solution might be a pilot study with a similar population that asks people to indicate the extent of applause they expect from teachers/students who show full support of a graduate. I could then use the results from the pilot study for rescaling in the main study. $\endgroup$
    – Flo
    Sep 23, 2016 at 16:42

1 Answer 1

0
$\begingroup$

I would question whether rescaling the teacher ratings is appropriate in all cases because it's not entirely clear whether the teachers are, or more critically whether they're perceived to be, rescaling their responses.

As an example, let's say I can feel supportive towards somebody on a scale of 0-100%. I can applaud at volumes ranging 0-100dBs. If I feel 0% support then I'll produce 0dB applause, feeling 50% support then 50db applause, etc. Now, let's say that I'm not allowed to produce more than 75dbs. I have two options. I could keep my scale the same so if I support someone 75% then they get 75dBs, but also if I support someone 75-100% then I give them 75dB applause. Let's call this the filter hypothesis.

Alternatively, I rescale it so that 0% support gets 0dB applause, 100% support gets 75dB applause, and I rescale the 1-to-1 mapping so, say 50% support gets 37.5dB (half-way between 0 and 75dB). Let's call this the rescale hypothesis. I'm not sure that rescaling your data makes sense if the filter hypothesis is correct.

Unless raters reliably believe that the rescale hypothesis is correct (and potentially, unless it actually is correct) I'm not sure that you should rescale their responses.

The notion that an identical response to two questions (say 80/100) does not reflect the same amount of the latent construct (support for students) is not uncommon. In IRT, it is actually encouraged for questions to vary in difficulty and, subsequently, it's quite normal for responses to differ between questions whilst still providing a reliable measure of the latent construct. I'm not sure I believe that this will be particularly problematic in your case.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.