I am currently building species distribution models (SDMs) (or ecological niches) using machine learning algorithms to predict the potential spatial distributions of animal species based on environmental variables. In my case, my SDMs use presence data (coded as 1) and pseudo-absence data (coded as 0) as the response variable, with environmental variables as predictors. My dataset includes an equal number of presence and pseudo-absence data points. I would appreciate some advice on selecting the most suitable model performance measures in this context.

One study recommends not relying solely on AUC (Area Under the Curve) as a single performance measure for models, and suggests employing other performance measures, including sensitivity and specificity. Based on this information, I have prioritized sensitivity (1st choice), specificity (2nd choice), and AUC (3rd choice) in selecting the best model. Does the choice of these three measures and their order of importance seem reasonable? Or are there better measures in my context? Any advice would be greatly appreciated as I am new to SDMs.

Update I've seen that PR-AUC (Precision-Recall Area Under the Curve) is a more appropriate measure for unbalanced datasets than AUC. Is it also appropriate for balanced datasets (i.e., an equal number of presences and pseudo-absences)?

  • $\begingroup$ What do you mean by "I have prioritized"? Looking at sensitivity alone for sure isn't a good idea as sensitivity is maximised if everything is classified positive. The opposite holds for specificity, which therefore in effect has the same problem. There's a tradeoff between these two, so they should always be considered together. Question: Does your algorithm just give out presence/absence, or does it give you a probability for each case? $\endgroup$ Commented Jun 14 at 9:18
  • $\begingroup$ Am I right that in your situation it is a bigger problem to classify presences as absences than to classify pseudo-absences as presences, because the pseudo-absences are "pseudo", i.e., it may be that they are indeed presences; the animal is there but wasn't observed? $\endgroup$ Commented Jun 14 at 9:21
  • $\begingroup$ @ChristianHennig Thank you very much for your comments. I evaluate the model's performance using all three measures (so I don't rely on just one measure), but I rank the values of these measures based on their chosen importance: first sensitivity, then specificity, and finally AUC. The algorithm provides both presence/absence classification and a probability for each case. It is actually the probability for each case (habitat suitability) that interests me the most. I ensured that pseudo-absences are generated by environmentally constraining them to regions with lower suitability values. $\endgroup$ Commented Jun 14 at 19:39
  • $\begingroup$ You may want to have a look at proper scoring rules for evaluating probabilistic predictions (on the Wiki page go to "Examples for proper scoring rules", but you may read some stuff before). $\endgroup$ Commented Jun 14 at 21:51
  • $\begingroup$ @ChristianHennig Thank you for your suggestion. I've noticed there are many evaluation metrics available. However, I'm looking to determine the best ones for my specific context. If you have any specific recommendations, I'd be very interested. Thank you! $\endgroup$ Commented Jun 17 at 14:19


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.