How can I know how 'descriptive' my data is?

Question

I'm an undergrad learning about Statistical/Machine Learning and I've implemented and fitted some models now using methods such as Neural Networks and SVM. I've noticed that when testing some datasets I get more than 95% accuracy (Iris) and others 50% (Wine quality). However here's the thing:

• Iris has 150 samples and Wine has 4989 samples

My intuition says the as $n \rightarrow \infty$ then the function should be easy to fit and the distribution should become clear. However this is a nice counter example. I know I can test for correlation between features and outcomes, or perform some feature selection but I don't know if this answers my question.

So my question really is: Is there any way to quantify a priori how good or descriptive a dataset is?

Of course a priori means without using a model to fit it.

Answer 1

First of all, I'm assuming when you refer to accuracy you used cross validation to asses how good the model predicts the classification outcome. Even if your training sample size goes to infinity, it might be that your model is not able to correctly predict outcomes based on the features.

In the case of SVMs, remember a hyperplane is constructed that minimizes misclassifications (of the training set) and maximizes the decision boundary. For a 2d visual representation see: http://www.saedsayad.com/support_vector_machine.htm As more datapoints are added, we would expect our classifier to increase its performance up to a certain point. This depends on how well our model is capturing the variability present in the data. To quote George Box: "All models are wrong, but some are useful". You could imagine that if you have an infinite amount of samples but the features in the samples are not adequate to explain all the variation in the sample, we would still have misclassifications. In my opinion, this is probably what is happening in your example. The iris dataset probably contains features that explain the variability quite well and this ensure a good decision boundary (in SVM). The wine dataset might contain many samples that even after optimization of the decision boundary contain a lot of misclassifications. The comparison between your two datasets isn't fair because they can differ in their features. A better approach would be to take a huge dataset and first start with a small part of it to train a classifier. Gradually increase sample size and performance will most likely increase.

From my limited understanding of Neural Networks, I'm not sure your samplesizes mentioned here are adequate for such an algorithm. There are rules of thumb, depending on the number of layers etc. As you didn't mention what algorithm or setup was used for the Iris or Wine dataset, a quick reference can be found here: ftp://ftp.sas.com/pub/neural/FAQ3.html#A_hu

Now onto your question. If by descriptive you mean how well the sample (that is your dataset) represents the total population from which it is drawn, you need more samples to check this. If you meant how descriptive my dataset is (in this case, how well does it classify) without actually training the algorithm and testing it, that would be really difficult exept for very simple datasets. If we go back to the two dimensional SVM example I linked, a simple scatterplot with labels for the two classes gives you a good idea how well a binary linear classifier would preform. As soon as we enter a higher dimensional feature space or have many possible outcomes, things get difficult. The above mentioned is just a visual inspection of the data, I can't think of any quantifiable way. But this leads me to the question, why would you want this in the first place?

I hope I've been some help to you, feel free to ask more if something isn't clear.

Answer 2

In a way, you are asking about something similar to effect sizes, which is defined as:

$$ d = \frac{(\mu_1 - \mu_2)}{\sigma} $$

where $\mu_1$ is the mean of treatment 1, $\mu_2$ is the mean of treatment 2, and $\sigma$ is some (pooled) measure of spread. This concept is straightforward if you know there are two groups and you know the group assignments. In such cases, you can determine if there are significant effect sizes. If the effect size (independent of your model) is large, then you will have an easier time because the dataset is "easy".

The problem arises when you do not know the means $\mu_1$ and $\mu_2$ (and the scale), necessitating their estimation so that you estimate the effect size with $\hat{\mu}_1$ and $\hat{\mu}_2$. If you do not know the true (unconfounded) assignment of instances to one of the means, or if your means are conditional not in treatment 1/2 but conditional on continuous high-dimensional features $x$, and you have uneven sampling across $x$, then you are already in trouble. This is because your effect size might vary with $x$:

$ d(x_1, x_2) = \frac{\mu(x_1) - \mu(x_2)}{\sigma(x_1, x_2)} $

Your dataset might have a good, large effect size $d$ for some feature "areas" $\{(x_1, x_2)\}$ but a bad, small effect size $d$ for other feature areas.

This might be especially problematic if you use the drop-in replacement $\hat{\mu}$ instead of $\mu$ for estimating $\hat{d}$ because then additionally you might encounter

• poor sampling of the data you collected,
• confounding (some important features not showing up in $x$ although you should control them),
• and so forth.

As a result, when estimating the effect size, you might never know whether a dataset is "easy" or not.

To practically address this question, you can test your predictive performance in feature areas $\{(x_1, x_2)\}$ that are important to you. On a holdout set or resampled holdout sets, you could:

A) Check if your model $\hat{\mu}(x_1) - \hat{\mu}(x_2)$ can predict the ground truth differences $\mu(x_1) - \mu(x_2)$ well enough for your application.

B) Compute the mean squared error (MSE). If your model's prediction is always on the spot, you might get a small MSE.

Practically, I tend to apply a baseline model which has performed well in similar tasks (e.g., XGBoost, LightGBM) on the dataset. If these models cannot find a reliable pattern in holdout test sets, your dataset might not contain enough information. Be cautious, though, as you need to choose a model with enough capacity to learn the patterns you seek. A rule of thumb is: if some independent humans can find a pattern in your dataset (e.g., "we know if this and that happens, then this sensor was high"), you have a "good" problem.