Many psychological studies involve getting participants to answer a hundred or more closed ended questions. A standard context would be a personality test with 100 items where each item is answered on a 1 to 5 scale. Items are designed to measure various scales and items vary in whether they are positively or negatively worded.

I often want to quickly identify participants who have answered the test randomly or in some other problematic way. I don't want to remove outliers in the purely statistical sense. For example, participants who are just very low or very high on the psychological scales might be flagged as extreme by some multivariate distance measures. I want to remove participants who have not completed the test conscientiously (e.g., random responding).

In the online environment, item response times can be very effective in identifying item skippers. However, assuming you only have item responses for a sample of participants:

  • What is a good basic procedure for flagging potential random responders?
  • Once such cases have been identified, what is a good strategy for determining whether they are random responders or just a bit unusual?
  • Are there any simple functions in R that implement the proposed approach?
  • 2
    $\begingroup$ I often want to quickly identify participants who have answered the test randomly or in some other problematic way. A (negligent) participant is very rarely able to answer in a truly random manner. Some systematic "problematic way" is almost present. Each such way has its own characteristic profile of answers. So, it is feasible to trace such profiles, at least most pronounced of its types. The problem however is to agree upon which profile shape/type/pattern is "problematic". $\endgroup$ – ttnphns Oct 29 '14 at 8:24
  • 1
    $\begingroup$ (cont.) There is a number of well-known questionnaire-filling effects (such as carryover, halo, positive pole bias, etc.) but one have carefully screen them to decide what is problematic and what is normal/acceptable in his study. $\endgroup$ – ttnphns Oct 29 '14 at 8:25
  • 1
    $\begingroup$ (cont.) A coefficient of a pronouncedness for each prototypic shape (for example, for jagged profile 1-5-1-4-2-5-1-5) can be invented and then, having distribution for the coefficient, 2-5% of utmost/outlying values could be discarded. $\endgroup$ – ttnphns Oct 29 '14 at 8:35
  • 1
    $\begingroup$ (cont.) As for truly random answering - if that occures at all - produces no bias in statistics. Correlations are weaker, dispersion are wider, reliability goes down. It's not as bad as the lost of validity, after all. I doubt that random answering can be tracked down. $\endgroup$ – ttnphns Oct 29 '14 at 8:44

Jeromy's own answer is more useful than mine, but since I am working on this myself, I have collected a few handy R code bits that can also be helpful or steer one in the right direction of detecting 'suspicious' responses.

The assumption here is that participants often (I find) respond in a pattern of some sort. Especially so in longer questionnaires.

For example, if participants choose the middle response category from top to bottom of a given scale, one you could check how many responses are unique in the scale (or even the entire survey) by using:

table(apply(data,1, function(X) {length(unique(X))}))

If you interested to know which cases in your data present this pattern, rather than the total per category, you can use this code:

 apply(data, 1, function(X) all(abs(diff(X)) == abs(1)))

Yet another way of looking at this is by at those participants whose responses alternate the response categories in diagonal lines (e.g., 1, 2, 3, 4, 5, 4, 3, 2, 1 or 2, 3, 4, 3, 2, 3, 4 which use just the center response categories.) For these respondents, one would need to calculate the lagged differences between responses with the following code:

apply(data, 1, function(X) all(abs(diff(X)) == abs(2)))

The careless R package on github includes a few nice functions.


Collection of Implementations for Indices of Careless Responding... [It] provides functions to support the computation of indices of careless responding. Initial indices include even/odd, psychometric synonyms, outlier, and longstring as described in Meade & Craig (2012)

Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological methods, 17(3), 437. PDF

Another package of the same name on github but with different code is available here: https://github.com/mattsigal/careless


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.