Here I have a question that might seem crazy, but I appreciate any thoughts and experiences:
When we have a dataset and a classification task at hand, we usually look for a Machine Learning (or in particular Deep Learning) method to extract and learn a set of decerning features from the dataset then train a classifier on it. The higher the accuracy of the classifier on test dataset, the better our proposed model.
Now, I am wondering about something opposite: Are there methods or approaches for assessing the quality of a dataset itself (and not a proposed model or algorithm) in being good to learn a set of decerning features from it for a specific classification task?
Just for clarification: consider MNIST dataset. We know that for the task of recognizing the digit, we can easily propose a model with more than 99% classification accuracy. But, if we label every sample in the MNIST dataset with a random binary label ( I mean half of them are labeled 0 and the rest are labeled 1 ) then it is near impossible to find a classifier for learning this binary classification task. The first reason is we cannot learn any feature from dataset to distinguish samples based on this binary label. In other words, there is no information in the data about the assigned label.
So, suppose we have labeled dataset, and someone gives us a classification task, and we feel there is no way to train a classifier for this wanted task with this dataset. Is there any method to support this feeling?