1
$\begingroup$

I'm an undergrad learning about Statistical/Machine Learning and I've implemented and fitted some models now using methods such as Neural Networks and SVM. I've noticed that when testing some datasets I get more than 95% accuracy (Iris) and others 50% (Wine quality). However here's the thing:

  • Iris has 150 samples and Wine has 4989 samples

My intuition says the as $n \rightarrow \infty$ then the function should be easy to fit and the distribution should become clear. However this is a nice counter example. I know I can test for correlation between features and outcomes, or perform some feature selection but I don't know if this answers my question.

So my question really is: Is there any way to quantify a priori how good or descriptive a dataset is?

Of course a priori means without using a model to fit it.

$\endgroup$
0
$\begingroup$

First of all, I'm assuming when you refer to accuracy you used cross validation to asses how good the model predicts the classification outcome. Even if your training sample size goes to infinity, it might be that your model is not able to correctly predict outcomes based on the features.

In the case of SVMs, remember a hyperplane is constructed that minimizes misclassifications (of the training set) and maximizes the decision boundary. For a 2d visual representation see: http://www.saedsayad.com/support_vector_machine.htm As more datapoints are added, we would expect our classifier to increase its performance up to a certain point. This depends on how well our model is capturing the variability present in the data. To quote George Box: "All models are wrong, but some are useful". You could imagine that if you have an infinite amount of samples but the features in the samples are not adequate to explain all the variation in the sample, we would still have misclassifications. In my opinion, this is probably what is happening in your example. The iris dataset probably contains features that explain the variability quite well and this ensure a good decision boundary (in SVM). The wine dataset might contain many samples that even after optimization of the decision boundary contain a lot of misclassifications. The comparison between your two datasets isn't fair because they can differ in their features. A better approach would be to take a huge dataset and first start with a small part of it to train a classifier. Gradually increase sample size and performance will most likely increase.

From my limited understanding of Neural Networks, I'm not sure your samplesizes mentioned here are adequate for such an algorithm. There are rules of thumb, depending on the number of layers etc. As you didn't mention what algorithm or setup was used for the Iris or Wine dataset, a quick reference can be found here: ftp://ftp.sas.com/pub/neural/FAQ3.html#A_hu

Now onto your question. If by descriptive you mean how well the sample (that is your dataset) represents the total population from which it is drawn, you need more samples to check this. If you meant how descriptive my dataset is (in this case, how well does it classify) without actually training the algorithm and testing it, that would be really difficult exept for very simple datasets. If we go back to the two dimensional SVM example I linked, a simple scatterplot with labels for the two classes gives you a good idea how well a binary linear classifier would preform. As soon as we enter a higher dimensional feature space or have many possible outcomes, things get difficult. The above mentioned is just a visual inspection of the data, I can't think of any quantifiable way. But this leads me to the question, why would you want this in the first place?

I hope I've been some help to you, feel free to ask more if something isn't clear.

$\endgroup$
  • $\begingroup$ Yes I used cross validation and the accuracy is similar between the models. To answer your question, I was thinking along the lines of "If I get a really low accuracy for a given dataset, is it my model's fault or the dataset's?" The use is to assess the dataset, not the model. This leads me to questions like "What is a good dataset?" or "What properties make datasets hard to fit?" $\endgroup$ – Carlos Brito Jun 23 '16 at 16:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.