Is there a way of quantifying something like an AUC per observation. What I'm trying to estimate is how easy each observation is to classify. So for example, given a test set $X$ of size $m$ containing observations (rows) ${x_1, ..., x_m}$ and a trained model $M$, I want to quantify how easy it is to for the model to correctly classify $x_i$ for each $i \in 1...m$ either on an absolute scale or relative to the other $x_j$, $j \neq i$. Here, $x_i$ is just a row containing values for a some set of features.

  • $\begingroup$ This is a very crude solution, but, if it's a two-class classification, how about just getting a vector predicted class probabilities from the model and then calculating: abs(predicted_probability - 0.5)? High values will indicate the observations that the model was sure about, and low values will be the ones that were close to 50/50. If you want to include information about whether the model was right or wrong, then multiply the values by 1 if it was right and -1 if it was wrong. $\endgroup$ – Adam B. Feb 3 at 22:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.