1
$\begingroup$

I am trying to transform my experiment results into variables and I am having some issues.

Data:

One likert scale 1-9 (

  • 1 = " I am sure I havent seen this before”
  • 5 = "Not sure”
  • 9 = “I am sure I’ve seen this again”

So, I was measuring the certainty of the response.

However, the response had also a right or wrong answer, which affects the accuracy of the response.

Accuracy is 0 or 1 (0 = false, 1 = correct). Participants who answered 1-4 but actually saw the image before were labeled as wrong, and those 6-9 were labeled as right. The opposite for those images presented only the second time.

(I realised I shouldn’t be using a likert scale for this purpose, as its meant to be for opinions and not right/wrong responses).

Although I could only use the accuracy of response as my dependant variable, I was thinking that I should utilise the certainty as well. I am thinking about transforming my likert scale as :

1 = 4

2 = 3

3 = 2

4 = 1

5 = 0

6 = 1

7 = 2

8 = 3

9 = 4

And accuracy as:

0 = -1

1 = 1

Then multiply both variables to get my final accuracy with certainty, with a range of -4 to 4, using certainty as a weight.

Would this approach be a correct way of dealing with my example ?

$\endgroup$
1

4 Answers 4

1
$\begingroup$

You have initial responses $X$ taking integer values 1 through 9.

Then you have transformed 'certainty' scores $Y,$ defined as $Y = |X - 5|,$ taking values 0 through 4. I think these should be scores $Y = X - 5$ taking values -4 through +4.

Then you should multiply by 1 or -1 to get scores $Z$ from -4 through 4, so that negative scores correspond to various degrees of wrong answers and positive scores correspond to degrees of correct answers. And finally, you would average these final scores $Z$ to get the overall rating for a subject.

Let's see how this would work for three kinds of subjects, Clueless, (mainly) Attentive, and (almost) Perfect. Specifically,

  • Clueless subjects answer with an initial response 1 through 9, equally likely and at random;

  • Attentive subjects assign initial scores 1 to 6 to images not seen before and scores 4 through 9 to images seen before (mainly correct or uncertain, occasionally wrong);

  • Perfect subjects initially give 1 or 2 to images not seen before and 8 or 9 to the other images (always correct, but with slightly varying degrees of certainty).

In the simulations below, I suppose that each subject sees 50 recycled images and 50 fresh ones, for a total of 100 images. Also, 10,000 subjects of each of the three kinds are simulated. Summaries of averages scores are found and graphed as histograms. [The simulation is written in R. I have used loops instead of defining functions (perhaps more elegant), hoping that non-R users can follow the logic.]

# Clueless
set.seed(725)
m = 10000;  n = 100;  s.c = numeric(m)
t = rep(c(-1,1), each=50)
for(i in 1:m) {
  x = sample(1:9, n, rep=T);  y = x-5
  z = t*y;  s.c[i]=mean(z) }
mean(s.c);  sd(s.c)
[1] -0.00221
[1] 0.2579402
summary(s.c)
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
-0.98000 -0.18000  0.00000 -0.00221  0.17000  0.97000 

.

# Attentive
m = 10000;  n = 100;  s.a = numeric(m)
t = rep(c(-1,1), each=50)
for(i in 1:m) {
  x1 = sample(1:6, n/2, rep=T);  x2 = sample(4:9, n/2, rep=T)
  x = c(x1, x2);  y = x-5
  z = t*y; s.a[i]=mean(z) }
mean(s.a);  sd(s.a)
[1] 1.498803
[1] 0.1709028
summary(s.a)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  0.850   1.380   1.500   1.499   1.620   2.170 

.

# Perfect
m = 10000;  n = 100;  s.p = numeric(m)
t = rep(c(-1,1), each=50)
for(i in 1:m) {
  x1 = sample(1:2, n/2, rep=T);  x2 = sample(8:9, n/2, rep=T)
  x = c(x1, x2);  y = x-5
  z = t*y;  s.p[i]=mean(z) }
mean(s.p);  sd(s.p)
[1] 3.499701
[1] 0.04965146
summary(s.p)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.  
   3.26    3.47    3.50    3.50    3.53    3.68 


par(mfrow=c(1,3))
  hist(s.c, prob=T, col="skyblue2", main="Clueless")
  hist(s.a, prob=T, col="skyblue2", main="Mainly Attentive")
  hist(s.p, prob=T, col="skyblue2", main="Almost Perfect")
par(mfrow=c(1,1))

enter image description here

With the slight modification made above, I think your scoring system would work fine. It certainly distinguishes well among the three imaginary types of subjects I considered.

$\endgroup$
0
$\begingroup$

Yes, that is OK.

Another option would be to recode answers for images that was actually not seen by participants before. In those cases you can recode 1 to 9, 2 to 8, 3 to 7 and so on (or just take $10-x$, where $x$ is original answer).

This would results in scores 1 to 9 for each question. If you prefer -4 to 4 scale, just subtract 5.

$\endgroup$
0
$\begingroup$

What an excellent and intriguing question!

I would propose a different kind of transformation, though, that retains a little bit of more information.

What you basically have in your survey is a human (/ are a bunch of humans) trying to predict a ground truth (= "seen before" / "not seen before") that is known only to you and you alone. For all those predictions those humans have offered a self-reported confidence level (ranging from "sure" [1,9] to "not so sure" [4,6]).

I'm heavily into machine learning projects right now (and since if you show a problem to an apple tree it would always propose "apples, of course!" as the solution) I immediately thought of a confusion matrix with a variable confidence level as a possible approach.

How would that work?

  1. Transformation / Feature Engineering
  2. Building the Confusion Matrix
  3. Scoring the Test

1. Transformation / Feature Engineering

Assumptions:

  • We could assume that answers with a Likert Scale value from 1 to 4 predict a "0" and answers with a value from 6 to 9 predict a "1".
  • We could also assume that there are the following confidence levels:
    • 4 and 6 — 25% confidence (not so sure)
    • 3 and 7 — 50% confidence (a bit sure)
    • 2 and 8 — 75% confidence (moderately sure)
    • 1 and 9 — 100% confidence (sure)
  • We could also assume that an answer with a value of 5 can be treated as "N/A", as the participant was not sure at all about their answer.

Therefore we can construct a table from the answers like

RespondentID, QuestionID, GroundTruth, Prediction, Confidence 1702,1,1,1,0.75 1702,2,0,1,0.25 1702,3,1,1,0.75 1702,4,1,1,1.00 1702,5,0,1,0.50 1702,7,1,1,0.25 ...

We also can calculate a result column with values being either 0 or 1 (i.e., was the answer correct?) from GroundTruth and Prediction.

This would look like this:

RespondentID, QuestionID, GroundTruth, Prediction, Confidence, Result 1702,1,1,1,0.75,1 1702,2,0,1,0.25,0 1702,3,1,1,0.75,1 1702,4,1,1,1.00,1 1702,5,0,1,0.50,0 1702,7,1,1,0.25,1 ...

Now we can calculate: How good was our participant, actually?


2. Building the Confusion Matrix

A confusion matrix a fundamental tool in the evaluation of predictive systems. Your participants, for the purpose of this experiment, is such a system. So let's evaluate them! There are four central terms you have to know:

  • True Positive - Has seen the picture before and answered correctly
  • True Negative - Has not seen the picture before and answered correctly
  • False Positive (False Alarm) - Has not seen the picture before and falsely remembered it
  • False Negative (Miss) - Has seen the picture before, but could not remember

True Positive and True Negative are good cases, False Positive and False Negative are bad cases. With this information you can actually find out how good the participant performed on various metrics. For this purpose, I have created an Excel file with dummy data so you can play around with it.

https://drive.google.com/file/d/0B4QXhnd6LLcHaEl6SjkwN0NvSVhVc3VhRTNVT0NxeXhPaE9v/view?usp=sharing

So, but how do we calculate a score?


3. Scoring the Test

You have already proposed a method: using the confidence as a multiplier. There is nothing wrong with it, buuuuut:

  • Was this methodology clear from the beginning?
  • Was it communicated clearly to the participants?
  • What does "sure" / "not so sure" mean for every participant? Are the meanings homogenous?

What is more, you should ask yourself what you want to achieve with your methodology:

  • Do you want to reward participants who are more confident? If yes, how much?
  • Do you want to punish wrong answers? If yes, how severely?
  • What do you want to do with participants who answered "5"?
  • and many more...

I mean, after all you could score the Matthews Correlation Coefficient (MCC) of the participant's answers?!

But just for completeness' sake I have added a points column, so it now reads

RespondentID, QuestionID, GroundTruth, Prediction, Confidence, Result 1702,1,1,1,0.75,1,0.75 1702,2,0,1,0.25,0,0.00 1702,3,1,1,0.75,1,0.75 1702,4,1,1,1.00,1,1.00 1702,5,0,1,0.50,0,0.00 1702,7,1,1,0.25,1,0.25 ...


All right, I have to go back to work. Hope I could give you some food for though. If you have any follow-up questions please do not hesitate to use the comment function.

Link to Excel file: https://drive.google.com/file/d/0B4QXhnd6LLcHaEl6SjkwN0NvSVhVc3VhRTNVT0NxeXhPaE9v/view?usp=sharing

$\endgroup$
0
$\begingroup$

This is not really an answer but rather an extensive comment.

  • What are you after, what is your dependent variable, what is your design?

  • You probably have not only correct and incorrect, but rather true positives, true negatives, false positives, and false negatives. This is more informative, and you may use measures such as hit rate, d', ROC curves, etc.

  • You loose information by combining accuracy and confidence into one measure. Making a combined model of both measures is much more complicated but will give you more information.

  • Cognitive psychologist often have similar data (accuracy + confidence) and use models from theory or multinomial processing tree models (e.g., Heck et al., 2018; Erdfelder et al., 2009).

  • Confidence rather than certainty might be a better term for googling.

Heck, D. W., Erdfelder, E., & Kieslich, P. J. (2018). Generalized Processing Tree Models: Jointly Modeling Discrete and Continuous Variables. Psychometrika, Advance online publication. doi:10.1007/s11336-018-9622-0.
Erdfelder, E., Auer, T.-S., Hilbig, B. E., Aßfalg, A., Moshagen, M., & Nadarevic, L. (2009). Multinomial processing tree models. Zeitschrift für Psychologie / Journal of Psychology, 217, 108–124. doi:10.1027/0044-3409.217.3.108

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.