0
$\begingroup$

Here I have a question that might seem crazy, but I appreciate any thoughts and experiences:

When we have a dataset and a classification task at hand, we usually look for a Machine Learning (or in particular Deep Learning) method to extract and learn a set of decerning features from the dataset then train a classifier on it. The higher the accuracy of the classifier on test dataset, the better our proposed model.

Now, I am wondering about something opposite: Are there methods or approaches for assessing the quality of a dataset itself (and not a proposed model or algorithm) in being good to learn a set of decerning features from it for a specific classification task?

Just for clarification: consider MNIST dataset. We know that for the task of recognizing the digit, we can easily propose a model with more than 99% classification accuracy. But, if we label every sample in the MNIST dataset with a random binary label ( I mean half of them are labeled 0 and the rest are labeled 1 ) then it is near impossible to find a classifier for learning this binary classification task. The first reason is we cannot learn any feature from dataset to distinguish samples based on this binary label. In other words, there is no information in the data about the assigned label.

So, suppose we have labeled dataset, and someone gives us a classification task, and we feel there is no way to train a classifier for this wanted task with this dataset. Is there any method to support this feeling?

$\endgroup$
  • 1
    $\begingroup$ Sure, just train every classifier, including the ones that haven't been invented yet, on the data set and determine that they don't work. $\endgroup$ – Sycorax says Reinstate Monica Jun 25 '18 at 14:45
  • 1
    $\begingroup$ @Sycorax Even if that were possible, it would surely result in overfitting the choice of model to the dataset. $\endgroup$ – Kodiologist Jun 25 '18 at 17:56
2
$\begingroup$

When you're wondering whether a variable in a dataset can be accurately predicted with other variables, there's no real shortcut for checking and finding out. If you try several different models and none of them are particularly accurate, and you don't know of any other important patterns in the data that the models weren't able to exploit, then you can reasonably conclude that one can't expect to do better. It is unlikely, although possible, that someone else will find a way.

$\endgroup$
  • $\begingroup$ At least when by accident having overfitted model. $\endgroup$ – Analyst Jun 25 '18 at 20:29
1
$\begingroup$

One thing would be to check if the distributions of values of different variables produce any information. Suppose that range of variation is limited in multidimensional space.

This kind of limited information about variation type of variable elimination could produce result where original many hundred variable data set contains at last only few informative.

Many classical methods would also fail in cases where variables in data set are highly correlated or values associated with other values.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.