1
$\begingroup$

I have a survey using a Likert scale and two inverted questions out of twenty. Using R how can I identify (and maybe filter out) the respondents that always tick agree, for example, and thus did not read the questions properly? Thanks.

$\endgroup$
1
$\begingroup$

I would recommend inverting the scale for both questions. So afterwards, you have two new variables where the polarity is inverted (e.g. before 6 = bad now 1 = bad). Then you can simply iterate of the variables and watch for people who have always 1 or 2 etc.... Additionally, the inverted scales are much better to analyze afterwards - no headaches because 2 out of 10 variables have to be interpreted "ass backwards".

As an additional note: Finding such people is remarkably difficult. Maybe the people are ticking in diagonal patterns or semi-randomly? There are programs out there who are generating random answers (e.g. ticking a random box out of 6 each time...) - so the question gets even more complicated.

So to filter out bad answers I would suggest your approach + look for the average answer time, candidates with very short times are prime candidates to sort out + check for very short open answers in unclear cases...

$\endgroup$
  • $\begingroup$ As a side note: If you really have inverted questions, I would not check if the people cliked agree - but if they checked "do not agree". If they clicked only without reading, they would never see that the question has a different meaning... $\endgroup$ – Christian Sauer Mar 19 '14 at 10:54
1
$\begingroup$

First, I preach caution in removing responses for this reason alone. It's possible to hold legitimately ambivalent attitudes on very many subjects, especially when negatively phrased items are not exact opposites of other, positively phrased items. For example, here are three items I work with a lot:

  1. "I have a good sense of what makes my life meaningful."
  2. "My life has no clear purpose."
  3. "My life has a clear sense of purpose."
    (Steger, Frazier, Oishi, & Kaler, 2006)

I wouldn't feel justified in excluding a person who rated both the first and second items Somewhat True; life can be meaningful without a clear purpose. Yet if a person rated both the second and third items Absolutely True, this might be reason to treat those two responses as missing. I still wouldn't throw out the other responses on this basis alone though. Where to draw the line between these two example cases is probably a judgment call (or at least a tough methodological question) that depends on many circumstances you haven't described, including your plans for the data after validating them.

I've used exclusion criteria in my work based on response invariance, but I set the bar somewhat lower so that only the worst cases failed to clear it. I.e., only completely invariant responding got a person's responses completely disregarded for a given measure with at least ten items that included at least one negatively phrased item or two subscales. This was easy enough to catch: I just analyzed a subset of my complete dataset that excluded participants with $SD=0$ across all items of a given measure. This might be a clumsy way of doing things, but it works – for example:

for(i in 1:length(MLQ[,1])){MLQ$MLQsd[i]=sd(as.numeric(MLQ[i,paste0('MLQ',1:10)]),na.rm=T)}

This line adds a MLQsd column to a data frame MLQ with ten columns named MLQ1MLQ10. Each row's value for MLQsd is the $SD$ of that row's values for those other ten columns. Anyone who gives the same response to all ten items including the negatively phrased item has a MLQsd $=0$. Excluding them is then as simple as using subset(MLQ,MLQsd>0).

Depending on how your data are formatted, you can modify the above code to generate other criteria for subsetting. E.g., assign each row a binary code: 1 if all responses =='agree', and 0 otherwise. Then just subset(your data, your binary code==0) if you want to exclude them. If you do, it's probably wise to estimate and report the size of the difference between excluded and retained observation groups. I also second @ChristianSauer's recommendation to look at response time if you have that data. I collected data on Qualtrics, and used completion time as another exclusion criterion.

Reference
Steger, M. F., Frazier, P., Oishi, S., & Kaler, M. (2006). The meaning in life questionnaire: Assessing the presence of and search for meaning in life. Journal of Counseling Psychology, 53(1), 80–93.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.