2
$\begingroup$

I have a dataset of the performance of probands in an experiment. An observer watching the performance noted positive and negative performance remarks on a simple tally, and ended up with something like

Proband ID  Positive Remarks  Negative Remarks
         1                 5                 2
         2                12                 8
         3                 6                 0

(assuming that each remark has equal weight.)

I would like to rate probands based on a combination of positive and negative remarks to be able to state a mean / median performance, to check for peculiar deviations and outliers, and to ultimately test for dependencies on other variables.

Intuitively, I would use a simple relation of Positive Remarks / Negative Remarks. Eyeballing this looks promising, but there are lots of zeros in the negative remarks.

How could I combine these variables into a single performance rating preferrably expressing the relation of positive to negative remarks in the presence of zeros?

I have checked related questions:

$\endgroup$
3
  • $\begingroup$ Assuming that not both Pos & Neg are zero, you could scale by the L1 norm. Scaled = P/ (P+N) Anything that is purely positive will get the score 1. Anything that is purely negative will get the score 0. Equal P & N gets 1/2. $\endgroup$
    – G5W
    Commented Jan 2, 2017 at 19:37
  • $\begingroup$ I would agree with @G5W, but want to interpret it as probability. Is that correct for your application. eg is there only positive and negative (or are there also eg don't knows?). Then you could consider confidence intervals on prob(positive) [so 2 performance ratings]. This deals with 'outliers' because of few number of remarks en.wikipedia.org/wiki/Binomial_proportion_confidence_interval $\endgroup$
    – seanv507
    Commented Jan 2, 2017 at 21:38
  • $\begingroup$ @G5W, L1 norm scaling worked beautifully for me. If you make your comment into an answer, I'll gladly accept it. $\endgroup$
    – fbmd
    Commented Jan 6, 2017 at 13:23

1 Answer 1

2
$\begingroup$

A simple metric that could work for a problem like this is:

Scaled = P/(P+N)

Anything that is purely positive will get the score 1. Anything that is purely negative will get the score 0. Equal P & N gets 1/2. It ignores the length of the vector. Along the lines of what was suggested by @sean507, this is just the proportion of all answers that are positive.

$\endgroup$
1
  • $\begingroup$ Could you please add a hint (and probably a link) that P+N originates from the L1 norm? The "cab driver distance" pointed me to another way of how two variables can be related. $\endgroup$
    – fbmd
    Commented Jan 8, 2017 at 7:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.