0
$\begingroup$

I am working on trying to get a summary metric that summarizes the results of four models (M1:M5) and preferably ranges from zero to one, with one being the best model and zero being the worst model. I have a dataset with five models df the values in the df are counts of species within each of the categories (a-d):

a) True Positives= whenever a species is predicted to occur by the stacked models and is observed by the field collections,

b) True Negatives= whenever a species is predicted to be absent by the stacked models and was not detected by the field collections,

c) False Positives= whenever a species is predicted to occur by the models but was not detected by the field collections and

d) False Negatives= whenever a species is predicted to be absent by the stacked models but was detected by the field collections.

M1= c(6,2,1,1)
M2= c(3,2,3,2)
M3= c(10,0,0,0)
M4= c(1,1,6,2)
M5= c(0,0,0,10)
names=c("a","b","c","d")
df=cbind (M1, M2, M3, M4, M5)
rownames (df)=names

This metric should also be standardized by the number of species in the community (n) and ideally true negatives (b) and false negatives (d) should be weighted less heavily than true positives (a) and false positives (b). Right now I have:

(1/n)(a+(1/b))/c+(1/d)).

Obviously this fails whenever there are zeros in my dataset. Does anyone have a suggestion on how to fix this.

$\endgroup$

1 Answer 1

1
$\begingroup$

A common metric for this type of data is "proper" scoring rules, which calculate a value from each observation based on the prediction of the outcome and the actual outcomes. "Proper" rules aren't completely arbitrary but must satisfy certain conditions.

Wiki gives an introduction to them which is decent, and when I did a search for "scoring rule" here, I got over 200 hits.

$\endgroup$
2
  • $\begingroup$ Would a proper rule like the quadratic rule apply with the presented dataset? $\endgroup$
    – I Del Toro
    Oct 9, 2014 at 16:50
  • $\begingroup$ Shouldn't be a problem. The R package is named "scoring". $\endgroup$
    – JenSCDC
    Oct 9, 2014 at 23:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.