How to analyse these data?

Question

I am conducting an experiment investigating lineup accuracy and witness confidence.

A long story short: we want to know what the pattern of false positives, hits and misses on a lineup task are under different lineup conditions and how confidence may vary with/independently of accuracy. Logically, witness confidence may also be affected by the different conditions, and we'd like to know this as well.

The between subjects variables are: Gender (male, female), ethnicity (Asian, Caucasian), and lineup type (sequential- where people see each lineup member one at a time and make a decision about each one, and simultaneous- where people see all the lineup members and make a decision about whether they see the perpetrator or not)

The within subjects variables are: Photo type (same vs different photo of the person), lineup ethnicity (Asian vs. Caucasian lineups), confidence (5 levels of a Likert scale from 1 "not confidence at all" to 5 "extremely confident)

The dependent variable is accuracy in terms of hits, misses and false positives (these could be coded as 0 or 1?) and correct recognition (hits-false positives)

One of the problems is that we want to know the relationship between confidence and accuracy, which would necessitate that confidence is an independent variable, however we also want to know if the other variables might affect confidence (such as ethnicity or lineup type), so I'm having trouble figuring out the best way to analyse this data.

Does anyone have any answers for me? Someone suggested maybe logistic regression, but they weren't really sure. I'm really not used to dealing with categorical data, so am in need of help!

Answer 1

The relationship between confidence and accuracy might potentially be investigated by considering them as a multivariate response; that it, you could analyze the conditional correlation between the two, conditional on some set of predictors.

The difficulty with that is that accuracy is a proportion while confidence is a Likert-scale item, makes the usual multivariate analysis tricky. It might be possible to deal with the Likert item as a multinomial logit and deal with some multivariate binomial relating those with the accuracy variable.

Another possibility might be to look at partial least squares type models (but again, even if you treat the Likert item as interval, there's the problem of the binomial accuracy); yet another possibility is some Bayesian graphical model.

Answer 2

If you have only 2 possible outcomes you should use a logistic regression. From your question I gather you have >2 possible outcomes, hence you should use a multinomial logit model. It is basically the same as a logistic (binomial) but more outcomes. These are all examples of generalized linear models this book has hands on coverage of GLMs as well as classification/regression trees. I did an example fit of a logistic regression in an answer to this question, you might want to check it out.

A classification tree make sense here too if you don't believe the interaction between the factors are too complex and significant.

Here is example code of how you would fit a multinomial in R (in this code none of the factors will be relevant because I randomly assigned them but with your real data they might be):

gender= c("male", "female")
ethnicity= c("asian", "caucasian")
lineupType = c("A","B")
outcomes = c("FALSE POS","TRUE POS","MISS")
genderObs = sample(gender,400,replace=T)
ethnicityObs = sample(ethnicity,400,replace=T)
lineupTypeObs = sample(lineupType,400,replace=T)
outcomesObs = sample(outcomes,400,replace=T)
library(nnet)
mmodF = multinom(outcomesObs ~ (genderObs + ethnicityObs + lineupTypeObs)^2) # full model with inter
mmod = multinom(outcomesObs ~ genderObs + ethnicityObs + lineupTypeObs)  # only the factors no interaction
mmodW = step(mmodF) # remove noise factors (here it will just be a fixed probability)
predict(mmod,data.frame(genderObs = "female", ethnicityObs = "asian", lineupTypeObs = "B"))
predict(mmod,data.frame(genderObs = "male", ethnicityObs = "asian", lineupTypeObs = "B"))
predict(mmod,data.frame(genderObs = "male", ethnicityObs = "caucasian", lineupTypeObs = "A"))
predict(mmodF,data.frame(genderObs = "female", ethnicityObs = "asian", lineupTypeObs = "B"))
predict(mmodW,data.frame(genderObs = "female", ethnicityObs = "asian", lineupTypeObs = "B"))

Here is a classification tree in R (on the same data):

library(rpart)
ct = rpart(outcomesObs ~ genderObs + ethnicityObs + lineupTypeObs)
pct = prune(ct, cp=0.05) # pruned tree
predict(ct,data.frame(genderObs = "female", ethnicityObs = "asian", lineupTypeObs = "B"))
predict(pct,data.frame(genderObs = "female", ethnicityObs = "asian", lineupTypeObs = "B"))

Answer 3

Yes, logistic regression would work, but also classification trees. I don't think you need to worry about false positives. It seems that the "confusion matrix" the model produces will tell you what you are looking for in terms of false positives and false negatives