2
$\begingroup$

I was reading this article entitled: Usage-Based Vehicle Insurance: Driving Style Factors of Accident Probability and Severity (Korishchenko et all., 2019) [1], and watching the results section, I saw that, for the AUC of their ROC Curve, they get numbers below 0.7, which could be considered as poor result (here is a source for values: [2]), and by the other hand, they use the McFadden pseudo R2 which, according to McFadden (1977) [3], Lee (2013) [4] and Louviere (p.55) [5], a good result must be between 0.2 and 0.4.; but the results in the article are below 0.2, being between 0.01 and 0.04 in one model with base features, and between 0.03 and 0.05 in the other with the features plus acceleration.

The main problem with this is, in the results, being one model better than the other by McFadden, and being the measure of AUC better than by chance, the values are below validated values, which for me, makes the model not quite to be trusted in terms of prediction capacity, except if there is another value range for AUC and McFadden R2 where this kind of results are acceptable on a relaxed statistical context.

Should I trust these results in the article or there is something to be rescued?

References

[1] Korishchenko et all., Usage-Based Vehicle Insurance: Driving Style Factors of Accident Probability and Severity, 2019, source: https://arxiv.org/pdf/1910.00460.pdf

[2] The Area Under a ROC Curve, source: http://gim.unmc.edu/dxtests/roc3.htm

[3] McFadden, Quantitative methods for analysing travel behaviours of individuals, 1977, source: http://cowles.yale.edu/sites/default/files/files/pub/d04/d0474.pdf

[4] Lee, A Comparison of Choice-based Landscape Preference Models between British and Korean Visitors to National Parks, 2013, source: http://www.lifesciencesite.com/lsj/life1002/286_B01288life1002_2028_2036.pdf

[5] Louviere, Stated choice methods analysis and application, source: https://www.researchgate.net/publication/215666083_Stated_choice_methods_analysis_and_application

$\endgroup$
  • 2
    $\begingroup$ It doesn't make any sense to speak of good values for AUC or Pseudo R squared free from the context of which outcome is being predicted, with what what predictors and for what purpose. You can simulate data where some outcome $Y$ and is connected to $X$ such that the the true model achieves a pseudo R squared of 0.04. You could totally trust this model to deliver its modest amount of predictive power. Whether such amounts of predictive power are useful or interesting depends on the context. Would be huge for whether stocks go up . Would be uninteresting for whether it will rain tomorrow. $\endgroup$ – CloseToC Feb 18 at 20:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.