Learning Curves: Why do people use the holdout set method for Decision Trees and K-NN, but RMSE for Neural Networks?

Question

Based on this question and this link, it seems like people generally use RMSE to understand error and generate learning curves when analyzing a neural network model.

But it seems like people generally just use error rate (or fraction of false predictions) to understand error and generate learning curves when analyzing a decision tree or k-NN model.

Is this correct?

Or are RMSE and error rate equally applicable when analyzing decision trees, k-NN, and neural network models?

Or does RMSE lend itself better to the analyzation of neural network models than it does for decision trees and k-NN models?

Answer 1

Generally, the error function does not depend on the learning method you are using, but on the application of the method. Is an error with a high confidence more expansive? Are all the errors equal?

When doing classification, you can either estimate the error rate or the error on the probability to belong to a specific class. For the later, you need your model to be able to produce the probability estimates.

Note that you have various error functions for probability estimates, such as (in the case of binary classification) :

• MSE = $\sum_i(p_i-y_i)^2$,
• MAE = $\sum_i|p_i-y_i]$
• logarithmic loss = $\sum_i(1-y_i)\log p_i+y_i\log (1-p_i)$ and many others...

A fully grown single decision tree cannot produce probability estimates. KNN can produce probability estimates (but these are hard to obtain, or you can use the ratio : positive neighbours / number of neighbours).

Neural networks, on the other hand minimize a specific MSE, that can be considered as a probability estimate. Naive Bayes or logistic regressions, by construction, look for probability estimates.

This is why you KNN or single decision trees are commonly evaluated in terms of error rates. But they can both, with specific parameters/implementation, provide probability estimates.