I'm working with a dataset of 102 rows (tabular data), from which I'm using 91 for training and 11 for testing. I'm using data augmentantion through the addition of gaussian noise for the training set. Supposing that I did all the right preprocessing (no data leakage), it would be rigth/acceptable to a scientific publication to apply data augmentation to my testing set, so it can be greater than just 11 rows?
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
1
Data splitting is not stable, especially not with such a small sample size.
Data augmentation is typically used to show your model multiple variations of the original observations, all slightly perturbed. One use of this is to ensure the model learns robust representations of the input. A more controversial application is to oversample less represented classes. However, despite what the name might suggest, data augmentation does not actually give you new, independent observations.
Hence, when you want to express the performance on unseen data, it is unclear why data augmentation would have any benefit. You are not actually producing new examples of unseen data.
What you could do instead is use (multiple repeats of) cross-validation to assess the performance of the model. This way you do not rely on a single, small sample for testing.
answered Jul 14 at 9:39
-
$\begingroup$ I totally second the advise to use (repeated) k-fold CV rather than a single split. But augmentation is completely independent of that thought - CV test sets could be augmented just the same. $\endgroup$ Commented Jul 15 at 17:59
$\begingroup$
$\endgroup$
3
Frans Rodenburg already gave a great answer (I especially second the K-fold cross-validation). I just want to add two points to it.
First, if you're going to add Gaussian noise to your test set, all you'll (most likely) end up doing is hurt your test performance somewhat (how much depends on how much noise you add). Your "real" test performance will be what you get on the unperturbed items. So not only do you not get any new information from "augmenting" the test set, but you'll actually get a distorted, pessimistic view of reality.
Second, artificially creating more test data will mess with most statistical analyses you might want to apply to this test performance. For instance, suppose that your classification accuracy on the test set is numerically barely above chance-level. You want to test whether this difference is statistically significant. Even if it isn't significant with your original 11 test examples, it is guaranteed to "become significant" (erroneously) as long as you add enough copies of these 11 test examples. With noise, of course, you're not adding exact copies (as I said, they'll likely be slightly worse), but near enough. Fundamentally, the assumption that many tests make, and which you'd be violating, is the assumption of independence between observations.
(Edit: There are ways to correct for this, which take the dependence structure within your data into account, or simply remove it (the averaging I suggest below is a very straightforward method for this, for instance). The important point is that you cannot simply apply the same test(s) you would otherwise use, when presented with a sample of independent observations. More fundamentally, you need to ask yourself what information you'd be gaining by adding perturbed samples. There may be good reasons, but just generating more data isn't one of them - the perturbation itself has to be meaningful, so that the performance on the perturbed samples tells you something you couldn't have learned from the unperturbed test set.)
If there are certain augmentations that you feel are likely to be representative of the range of data you expect to find when you apply your model in practice, you could try those, but then I would recommend averaging the performance across all the perturbations of a given test example, before applying any stats.
answered Jul 15 at 7:39
-
$\begingroup$ Two thoughts: a) "hurt your test performance": i.e., such a bolsered error estimate penalizes test cases that barely ended up on the correct side of the class boundaries. OTOH, cases that barely end up on the wrong side, are less heavily penalized than without this augmentation. b) statistical tests will be OK as long as one takes into account that the effective test sample size is still (approximately) the number of original, independent test cases. The situation is really not different from situations with e.g. repeated measurements of the same case (which happens routinely in my field). $\endgroup$ Commented Jul 15 at 17:57
-
$\begingroup$ I'm not sure what you're trying to say with your first thought? Obviously your performance can only drop due to previously correctly classified items becoming misclassified. The point is, you can arbitrarily degrade your test performance by any desired amount by adding noise, so this only makes sense if there is some prior reason to do so (and this is not the same reason why we apply augmentations to he training data). $\endgroup$ Commented Jul 15 at 20:02
-
$\begingroup$ As for your second thought: yes, of course if you model the dependence structure, or correct for it (e.g., by simply averaging the dependent items, as I suggested), then you're fine. The trouble would arise if you naively applied the same test you would otherwise choose (e.g., a binomial test) across your augmented sample, which is an easy trap for an inexperienced practitioner to fall into. I'll clarify my answer a bit to reflect this nuance. $\endgroup$ Commented Jul 15 at 20:08
$\begingroup$
$\endgroup$
5
Such or similar techniques have been described in the literature, e.g.:
U. Braga-Neto and E. Dougherty: Bolstered error estimation, Pattern Recognition 2004,
DOI 10.1016/j.patcog.2003.08.017
uses a (Gaussian) kernel around data points for generalization error estimationM. Sattlecker et al.: Assessment of robustness and transferability of classification models built for cancer diagnostics using Raman spectroscopy, Journal of Raman Spectroscopy 2010,
DOI 10.1002/jrs.2798
Augment with noise that corresponds to certain influencing factors found with Raman spectra.(Oversampling for class imbalance in testing is not needed, instead the tested cases can be properly weighted)
So, there are good reasons to augment test data - IMHO particularly when there are artifacts or influencing factors of which it is well known how they perturb the input data, but e.g. there is no good removal technique.
it would be rigth/acceptable to a scientific publication to apply data augmentation to my testing set, so it can be greater than just 11 rows?
Augmentation may simulate certain perturbations and thus be used to assess the model's ruggedness wrt. that perturbation.
The number of independent cases will still stay 11 for most statistical purposes - i.e., the effective sample size will be much smaller than the number of rows in the test data matrix.
-
$\begingroup$ Oversampling is still useful in cases with extreme class imbalances, when the training algorithm subsamples the data. For instance, when training a DNN to diagnose a rare disease using stochastic gradient descent. If (say) only 100 out of your 10,000 X-ray scans have a tumor, and you train with a batch size of 64, then many of your batches will have no tumors, which can lead to poor learning since the stochastic gradient estimates from those batches will be badly off, and in general those estimates will have high variance. This is not solved by overweighting the rare class in the loss. $\endgroup$ Commented Jul 15 at 20:19
-
$\begingroup$ @RubenvanBergen: I clarified that I refer to testing, since that's the question here. Wrt training, I still see far more potential in augmenation than in oversampling - but there are sufficient questions and answers here discussing this. For your example, my main point would be to consider that discriminative classifiers are often not a well suited as Ansatz for rare disease diagnostics. Moreover, a total of 100 cases of the disease means that sensitivity (for the sake of back-of-the-envelope guesstimate) will have a variance uncertainty of roughly 10 %pts, and ... $\endgroup$ Commented Jul 17 at 9:32
-
$\begingroup$ ... while there are loss functions that are "better behaved" than a hard correct/wrong proportion, they cannot do miracles, and there will be a problem with high (variance) uncertainty in the loss function for a classifier with only 100 positive cases available. Poor choice of batch size is yet a different problem, IMHO. $\endgroup$ Commented Jul 17 at 9:32
-
$\begingroup$ Oh, and I'm not talking of overweighting the rare class in testing. I'm reneferring to correcting the class weights according to the relative class frequencies under application condidtions since training and test/verification/validation data a often do not reflect these. In my field, the training and test data will often be oversampled already compared to the actual application. (And almost always if the application is screening, and often for rare disease diagnostics, as opposed to, say applications for differential diagnostics) $\endgroup$ Commented Jul 17 at 9:36
-
$\begingroup$ Consider your example scenario: a rare disease by definition affects < 5/10000. The diagnostics leading to your X-ray being done thus already need to increase the pre-test (before X-ray + DNN) odds by at least a factor of 20 in order to have 100/10000 data sets to represent the relative frequencies as representative for the application. This may be the case, but it is IMHO far from given. $\endgroup$ Commented Jul 17 at 9:43