What are acceptable validation or cross validation error rates?

Question

Is there a commonly acceptable error rate for validation? As in, if the error rate is less than X %, then my machine learning method would be considered "successful".

I'm looking for something analogous to a p-value of 0.05 used for many experiments, but for cross-validation.

I would use 5% as an error rate but thats really hard to achieve especially if you have small training and validation sets. (I only have 6 subjects total).

Answer 1

It's a bit hard to get a p-value below 5% when each sample represents 16.7% of the data ! But even with a very large sample size, there's no such thing as a "universal" acceptable error rate that would be suitable for all applications. The expected MSE of an estimator can be decomposed as bias^2 + variance + noise. So even a "perfect" learning machine will not allow you to get rid of the noise term which is application dependent. Intuitively, noise comes from the fact that the underlying data generating process is non-deterministic, i.e., $y = f(x) + \epsilon$, i.e., you may get different values of y (the target) for two samples with the exact same x (vector of inputs). The best (in MSE terms) predictor will be $\hat{y} = f(x)$ with error equal to $Var(\epsilon)$.

Answer 2

Very broadly speaking, there are two reasons someone would build an automatic classification system (a machine learner, if you will). Either you want to do better than an incumbent manual approach (such as a human expert or group of experts) or you want to do at least as well as the current approach, but use the automatic system to make the classification process faster and be able to handle large amounts of data.

The different applications then drive the acceptable error rate. If you are trying to improve the state of the art, then you need to demonstrate consistently better accuracy than what is currently achieved. Anything better than a benchmark is 'acceptable'. Alternatively, if you are hoping to automate some process currently done by a human then you might only need to do roughly as well as the benchmark since the benefits flow from the automation rather than accuracy boost.

I guess a third option is the case where there is no current system in place at all. In that case as suggested by Michael Chernick, an acceptable error rate could be derived by assessing the cost of misclassification compared to the utility gain from the successes. Again, the application the system is being built for is the driver, there are no absolutes.