4
$\begingroup$

I am a self-taught person and I would like your help. I am learning about predictive modeling in general, and I'm also trying to do predictive modeling for a specific problem.

I am exploring alternative ensemble learning algorithms (CART, Random Forest). For this problem and model, I am trying to predict three categories with arbitrary labels 0/1/2. I get a good value of accuracy (0.81), but very high error (Mean squared error = 0.37).

My question: How do I interpret these diagnostics? Is this a bad model? How should I interpret mean squared error? How should I define the intervals for mean squared error for an acceptable model?

$\endgroup$
4
  • 1
    $\begingroup$ It seems to me that the question you're asking in your post is different from the title of the post. Could you make the two consistent? $\endgroup$ Commented Feb 28, 2015 at 15:55
  • $\begingroup$ Thanks for ur comment! I am trying to predict a phenomena, I collected the features (based on the study I ve done) and defined the target variable (labeled it so it's three cases 0/1/2). When I used CART to predict, I found that accuracy is 0.81 and Mean squared error = 0.37. It appeared to me that the error is too high, can you explain how can I comment on this model, should I say that I have a good model (based on the accuracy), or should I say that I have bad model based on Mean squared error $\endgroup$ Commented Feb 28, 2015 at 19:05
  • $\begingroup$ Are the numbers 0/1/2 arbitrary labels to the categories or is there some sort of natural ordering? In other words, if you predict class 1 but its actually class 0 are you closer to correctly predicting than if you had predicted class 2? $\endgroup$ Commented Feb 28, 2015 at 22:34
  • $\begingroup$ yes, 0/1/2 are arbitrary labels to the categories. I just labeled desired results to numbers. $\endgroup$ Commented Feb 28, 2015 at 22:47

1 Answer 1

3
$\begingroup$

Since your labels for the three groups are arbitrary, your classification rate is more meaningful than Mean Square Error (MSE) and MSE ought to be ignored. MSE tells how you how far your classifications are from the true values. Since the values you are assigning your groups are arbitrary, there's no sense of how far or close a classification is from the true value. You've either correctly classified or you didn't and classification rate measures this.

You can see this for yourself - if you reassign the arbitrary labels to your three groups you'll see a different MSE result.

$\endgroup$
2
  • 1
    $\begingroup$ Why did you delete a similar answer to this and post this new one rather than edit the old? $\endgroup$
    – Glen_b
    Commented Mar 1, 2015 at 3:12
  • $\begingroup$ Because the OP made a comment that was no longer relevant. Im happy to reverse it. $\endgroup$ Commented Mar 1, 2015 at 6:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.