0
$\begingroup$

I am often dealing with online questionnaires and I have to identify respondents that do not fill out the questionnaire seriously in order to improve data quality.

In order to identify "speeders" who just fill in anything so that they can get the incentive at the end of the survey I analyse answering times (how much time was needed in order to answer each question). If answering time is below a certain threshold the person is marked as "speeder".

Now, I am trying to identify respondents who have suspicious answering patterns: Their answers are suspicious compared to the answers of the other respondents. So I need some kind of measure for how (dis)similar an answer is compared to the majority of the respondents.

For questions with a discrete number of answers, I can simply assign the frequency of the answer to the person: e.g. if there are three possible answers A, B and C and 90% of the respondents belong in category A or B, then persons who answered C will get a value of 0.1 etc.

At the end I can simply calculate the mean frequency over all questions. This implies of course that all questions have the same importance.

However, for answers on a continuous scale (e.g. body size) I still haven't figured out which method would be best. First, I thought I could simply use the distance from the median value. But if the distribution is bimodal, for example, this would not make much sense. So I thought I could simply "discretise" a continuous variable for example in 10 categories and then treat it as described above: If many respondents fall in the same category the similarity is considered to be high.

So my question is: Do you see any other possibilities to identify unusual answers? Is there maybe a standard?

#install.packages("trunknorm")
library(tidyverse)
library(truncnorm)

# Create some fake questionnaire data with different distributions
set.seed(100)

n <- 100
nn <- 1e4
testdf <- data.frame(var1 = round(rnorm(n, mean = 4, sd = 1), 0),
                     var2 = round(rnorm(n, mean = 3, sd = 0.7), 0),
                     var3 = c(rtruncnorm(nn/2, a=1, b=100, mean=30, sd=5),
                              rtruncnorm(nn/2, a=1, b=100, mean=60, sd=10)),
                     var4 = round(runif(n, 1, 30), 0)
                     ) %>%
  mutate(var1 = ifelse(var1 > 6, 6, var1),
         var1 = ifelse(var1 < 1, 1, var1),
         var2 = ifelse(var2 > 6, 6, var2),
         var2 = ifelse(var2 < 1, 1, var2)
)

A simple function to get the frequencies for all answers (an answer is considered to be "discrete", if it has <=10 categories)

checkSimilarity <- function(vector){
  
  categories <- length(unique(vector))
  df <- data.frame(value = as.factor(vector))
  
  # Categorical variables (less than 10 or 10 different categories)
  if(categories < 10){
    freqTable <- as.data.frame(table(vector)) %>%
      mutate(percent = 1 / sum(Freq) * Freq,
             vector = as.factor(vector))
    
    ret <- df %>% left_join(freqTable, by = c("value" = "vector")) %>% select(percent)
    
    
  # Continuous variables (more than 10 different categories)
  } else{
    ret <- NULL
  }
  
  return(ret)
}


checkSimilarity(testdf$var1)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.