This question already has an answer here:
- On the hardness of data to learn 2 answers
In a classification problem, the objective is to identify to which of a set of categories a new observation belongs, on the basis of a set of examples whose category membership is known.
The bottom-line of any solution to this problem is to find a set of features that describes the classification of the examples into the corresponding categories (hypothesis).
When the size of the set of categories is large, we need more information to distinguish among the different categories (e.g., we could look for more features). Also, if the examples for a particular category have high variance (intra-class variance), it is more difficult to find the discriminant due to the existence of collisions between members of different classes.
I'd like to know if there exists a quick exploratory method to evaluate how difficult a learning problem is. For example, I'd like to evaluate the number of collisions between different categories. The problem of training a classifier and performing some sort of cross-validation is that the solution is dependent on the choice of the learning model.