I have a question. Say you are given with like 1500 instances as your dataset, in which instance will be categorized into 1 of the 3 classes. You're supposed to come up with the best model from the best learning algorithm. Before that, you're supposed to have a default model for comparison in order to come up with some analysis, i.e. how is this best model better than the default model. What would be the default model for comparison in this case?
Thank you. Your help is very much appreciated :)
An obvious baseline to which your model can be compared is the model you have that uses no features, that is, a model that makes predictions just based on the categories.
Think this way: if I have a stack of $900$ dog photos mixed with $100$ cat photos, which animal would you predict is in a given photo if I did not let you see the photo? The costs of incorrect decisions would influence your decision, sure, but it is clear that it is much more likely that there is a dog in the photo than a cat. Consequently, predicting the majority category every time would be a viable strategy. In this case, you would score an impressive-looking $90\%$ accuracy without doing any kind of fancy modeling.
This is totally analogous to $R^2$ in regression problems. Further, it has intuitive sense. If you could score $90\%$ accuracy by making the same prediction every time, a model with $80\%$ accuracy, on its own, sounds like it has a solid score. However, using that model has doubled the error rate from $10\%$ to $20\%$. Thinking about good models, if you can score $90\%$ with the naïve model that predicts the "dog" category every time, and then you make a model that scores $99\%$ accuracy, that sounds like only a $9\%$ improvement, perhaps not very impressive. Thinking in terms of error rates, however, you have reduced the error rate from $10\%$ to $1\%$. That is, your model makes mistakes a tenth as frequently as it did before. The $R^2_{\text{accuracy}}$ I give in the above link would be $0.9$ for such a model, and this can be interpreted without the issues with classification accuracy that arise in problems when one category has more instances than another.
To your problem, you have three categories. Without even looking at the features, you could get a certain error rate by predicting the most common category every time. Your model better be able to do better than that and achieve a lower error rate.
I have a related answer here, and another answer to that question offers an interesting viewpoint that goes against my stance (granted, with a different sense of what is valued, which is not just the accuracy in that answer, so the two answers are perfectly compatible, depending on what is valued).
EDIT
All of the above assumes that you are interested in the categorical classifications. As is mentioned in one of the comments, there are reasons not to do this.
