Search Results

But in his article Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules Dr Harrel refers to these two as just proper scoring rules: The two most commonly used … proper scoring rules are the quadratic error measure, i.e., mean squared error or Brier score, and the logarithmic scoring rule... …

You need modifications to the bootstrap (.632, .632+) only because the original research used a discontinuous improper scoring rule (proportion classified correctly). … Improper scoring rules mislead you on the choice of features and their weights. In other words, everything that can go wrong will go wrong. For more see this. …

The proportion classified correctly is a discontinuous improper scoring rule. An improper scoring rule is one that is optimized by a bogus model. … With an improper scoring rule such things as addition of a highly important predictor making the model less accurate can happen. …

Proportion classified correctly is a discontinuous improper scoring rule that is optimized by a bogus model. I would not believe anything that you learn from it. …

Third, you have chosen as an accuracy score a discontinuous improper scoring rule (proportion classified correctly). Such an improper scoring rule will lead to selection of the wrong model. …

If by error rate you mean the usual one (proportion classified correctly), this is a discontinuous improper scoring rule. An improper scoring rule is optimized by a bogus model. …

The exact same issues also plague the F1 score (actually all Fβ scores), sensitivity, specificity and alternatives. Is there a standard academic article one can point to discussing these issues? … I have looked through Frank Harrell's "Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules". This kind of material is exactly what I am envisaging. …

Moving from continuous predictions, as used in computing ROC area (concordance probability; c-index) to a discontinuous improper scoring rule (forced-choice classification accuracy) results in all kinds … It is much better to make decisions on the basis of proper scoring rules (e.g., log-likelihood/deviance/logarithmic probability scoring rule; Brier score (quadratic probability accuracy score)). …

The cross-validation measures you seem to be computing are discontinuous improper accuracy scores. … They are designed not to agree with proper scoring methods such as likelihood-based measures and the Brier score. …

As Frank Harrell notes in this answer: You need modifications to the bootstrap (.632, .632+) only because the original research used a discontinuous improper scoring rule (proportion classified correctly … So I suppose that you could compute a .632+ score for other purposes, but there might not be much point. I suspect that accounts for any paucity of functions in R with respect to .632+ estimates. …

The problem is that proportional 'classified' correctly is an arbitrary discontinuous improper accuracy scoring rule. Don't trust anything you learn from it. …

I would highly discourage the use of any discontinuous improper scoring rule (an accuracy score such as sensitivity, specificity, proportion classified correct that when optimized results in a bogus model …

This is a classic example of the harm caused by the use of a discontinuous improper scoring rule. … Sensitivity and specificity are also improper accuracy scores and are discussed at length in the Diagnosis chapter of the same notes. …

The fact that you are bringing up the issue of balance means that you have not considered the fact that proportion "classified" "correctly" is a discontinuous improper accuracy scoring rule. … If you use a proper scoring rule (e.g., Brier score or pseudo $R^2$) the issue goes away. See this and this for more. …

Classification error is both discontinuous and an improper scoring rule. It has low precision, and optimizing it selects on the wrong features and gives them the wrong weights. …