3
$\begingroup$

I want to identify bad responses in the data we get from our vendors. To assess the quality of the data, what statistical tests can i apply to identify bad respondents?

I performed Discriminant analysis to identify the % similarity of responses among surveys completed by a particular interviewer. How can i further authenticate my findings?

(added from comments)

What i mean by bad responses is: Interviewers who have very high similarity in the responses they collect in the survey. they might tick for "4" for a particular question for all the respondents (Flat Liners or Speeders)

Suppose an interviewer has conducted 10 surveys and i notice similarity in the answers of those questionnaires for each question. Is there any statistical test which authenticates my finding that the interviewer has probably filled responses himself or has ticked the same response for each question? How can we check a particular data using Statistical tests? I hope i have clarified my question.

Improve this question
$\endgroup$
6
  • $\begingroup$ You need to tell us more about what you mean by "bad responses" and "bad respondents"; you also seem interested in "bad interviewers". $\endgroup$
    – Peter Flom
    Commented Feb 9, 2012 at 11:43
  • $\begingroup$ What i mean by bad responses is: Interviewers who have very high similarity in the responses they collect in the survey. they might tick for "4" for a particular question for all the respondents (Flat Liners or Speeders) $\endgroup$
    – Shantanu
    Commented Feb 9, 2012 at 12:22
  • $\begingroup$ Suppose an interviewer has conducted 10 surveys and i notice similarity in the answers of those questionnaires for each question. Is there any statistical test which authenticates my finding that the interviewer has probably filled responses himself or has ticked the same response for each question??? How can we check a particular data using Statistical tests?? I hope i have clarified my question.. PLEASE HELP... $\endgroup$
    – Shantanu
    Commented Feb 9, 2012 at 15:30
  • $\begingroup$ My first thought would have been to look at methods to detect cheating on multiple choice items, although methods to handle extreme response style or inter-rater analysis are likely candidates. $\endgroup$
    – chl
    Commented Feb 9, 2012 at 18:56
  • $\begingroup$ I'm not sure why you need a statistical test. If you suspect that the interviewer has ticked the same response for the same question across all interviews, then the sum or mean of responses for that interview can be examined. That will tell you absolutely that the same answer is chosen every time. Falsifying survey data requires another, trusted interviewer to interview the same respondent, so you have two results to compare. $\endgroup$
    – Michelle
    Commented Feb 9, 2012 at 19:22

3 Answers 3

Reset to default
3
$\begingroup$

The keywords to search for are "interviewer falsification". AAPOR/SRMS guidelines are a good starting place, and RTI system is useful, too. As far as I know the evidence, the interviewers may be able to get the first moments OK, but they have a more difficult time with higher order moments, so falsified data may be detected via unusual variances and correlations/crosstabs.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ (+1) That looks indeed like the right keyword to search for! $\endgroup$
    – chl
    Commented Feb 10, 2012 at 12:50
  • $\begingroup$ Hi...thanks for your suggestions. I am using discriminant analysis to group the responses that a particular interviewer gets for a question which basically tells me the % of similarity of the groupings. But this is countered by saying that the reason you are having similar responses is because your questions are similar. Now how do i prove that my questions are not similar. Is there any statistical procedure for this...thanks a lot all of you for your help... $\endgroup$
    – Shantanu
    Commented Feb 10, 2012 at 14:43
  • $\begingroup$ In psychometrics, somewhat similar ideas are covered under the umbrella of "divergent validity": if items are measuring something different, they should vary. That's again the second order moment issue of (co)variance between items. I am not even sure I see how you apply discriminant analysis here. $\endgroup$
    – StasK
    Commented Feb 10, 2012 at 19:48
1
$\begingroup$

I don't know of a statistical test that does what you want, although there may be one. Probably, though, you will need to define "bad interviewer" more precisely.

There is some literature on interviewer effects, and that might be a place to start looking.

Improve this answer
$\endgroup$
0
$\begingroup$

If your interviewers have completed more than 20 interviews you might consider the Wald-Wolfowitz test for randomness. Falsified data has a tendency to mix or cluster whereas randomly sampled data should exhibit a random order. Note that this only works for dichotomous variables.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ Precisely what would you be testing for "randomness"? At a minimum it seems your assertion about the "tendencies" of falsified data, in order to have some authority, needs to be circumscribed by a description of what range of applications you have in mind. For instance, responses to polls and surveys and responses to sets of related questions would normally exhibit "clusters" and degrees of similarity. $\endgroup$
    – whuber ♦
    Commented Mar 15, 2012 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.