Categorical data comparison

Question

I've been struggling with the following problem and I am not sure if I am using the right statistical method. Namely, I am analyzing passwords characteristics from a questionnaire and from real plane-text passwords. The answers from the questionnaire are categorized in 23 possible categories, regarding the password characteristics and the password composition: from category "1 - numeric - digits only" to category "23 - long sentence, digits and special characters". The plane-text passwords are also categorized in the same categories. My goal is to compare the two variables, the answers from the questionnaire and the plane-text passwords and to see if the answers from the questionnaire are in-line with the real passwords. Note that not all of the categories have a count > 0 (see pictures). Can you, please, tell me what is the right way to compare the answers and tell if there is some statistically significant connection between them?

Thanks for any suggestions.

Answer 1

If you want to compare sets of counts for different categories as a function of a grouping variable, you can do so with a chi-squared test. I don't have access to SPSS, but I can demonstrate the basic procedure in R.

(For what it's worth, I can't fathom your data. Let's assume your data are like this:)

tab = read.table(text="     4       31
        4        0
        0        0
        1        0
        0        0
        3        0
        1        0
        1        0
        1        0
        0        0
        0        0
        3        0
        2       53
        0        0
        0        0
        0        0
        1        0
        0        0
        1        0
        0        0
        0        0
        0        0
        0        0", header=FALSE)
tab                  = as.table(as.matrix(tab))
names(dimnames(tab)) = c("category", "password_type")
rownames(tab)        = 1:23
colnames(tab)        = c("questionnaire", "plain.text")
tab                  = tab[which(apply(tab, 1, sum)!=0),]
tab
#         password_type
# category questionnaire plain.text
#       1              4         31
#       2              4          0
#       4              1          0
#       6              3          0
#       7              1          0
#       8              1          0
#       9              1          0
#       12             3          0
#       13             2         53
#       17             1          0
#       19             1          0

Because you have many expected values <1, you cannot trust that the distribution of the test statistic will be well approximated by a chi-squared distribution. Therefore, you should simulate the sampling distribution, or perhaps use a permutation test.

chisq.test(tab, simulate.p.value=TRUE)
#  Pearson's Chi-squared test with simulated p-value (based on 2000
#   replicates)
# 
# data:  tab
# X-squared = 72.741, df = NA, p-value = 0.0004998