1
$\begingroup$

I have a Naive Bayes model that predicts 3 classes. As you increase each class it means that the condition is more severe. 0 means no condition, 1 is concern and 2 is that they have the condition. I have a model built but I am not sure how to correctly validate the accuracy.

Since 2 is further away from 0 it is less bad if a model classifies a 2 as a 1 instead of a zero. This means that my raw accuracy score is not completely correct.

Any help would be appreciated!

$\endgroup$
0
$\begingroup$

You could compare the inverse of the distance. Say we use the absolute value, meaning $distance(x,y) = | x-y|$.

The inverse of the distance from $0$ to $2$, e.g. $\frac{1}{|2-0|} = 0.5$, but from $1$ to $2$ is $1$. You just need to be careful when computing the ditance of a class to itself, setting it always to $0$, for example.

$\endgroup$
  • $\begingroup$ That makes sense when using that to validate, would I compute the distance from the actual - predicted and use that to validate? Wouldn't that just be an error term? $\endgroup$ – Dillon Lloyd Apr 3 at 17:02
  • $\begingroup$ It would just be an error if you're computing it for your training data. For a validation you simply compute it using you validation set .You can see my suggestion as a weighting for the mismatches, that gives a higher weight to the mismatches the user (you) consider more relevant. $\endgroup$ – Lucas Farias Apr 3 at 17:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.