# Should I use Mean Square Error or Classification Rate?

I am a self-taught person and I would like your help. I am learning about predictive modeling in general, and I'm also trying to do predictive modeling for a specific problem.

I am exploring alternative ensemble learning algorithms (CART, Random Forest). For this problem and model, I am trying to predict three categories with arbitrary labels 0/1/2. I get a good value of accuracy (0.81), but very high error (Mean squared error = 0.37).

My question: How do I interpret these diagnostics? Is this a bad model? How should I interpret mean squared error? How should I define the intervals for mean squared error for an acceptable model?

• It seems to me that the question you're asking in your post is different from the title of the post. Could you make the two consistent? Commented Feb 28, 2015 at 15:55
• Thanks for ur comment! I am trying to predict a phenomena, I collected the features (based on the study I ve done) and defined the target variable (labeled it so it's three cases 0/1/2). When I used CART to predict, I found that accuracy is 0.81 and Mean squared error = 0.37. It appeared to me that the error is too high, can you explain how can I comment on this model, should I say that I have a good model (based on the accuracy), or should I say that I have bad model based on Mean squared error Commented Feb 28, 2015 at 19:05
• Are the numbers 0/1/2 arbitrary labels to the categories or is there some sort of natural ordering? In other words, if you predict class 1 but its actually class 0 are you closer to correctly predicting than if you had predicted class 2? Commented Feb 28, 2015 at 22:34
• yes, 0/1/2 are arbitrary labels to the categories. I just labeled desired results to numbers. Commented Feb 28, 2015 at 22:47