1
$\begingroup$

So, I have a segmentation task with a dataset whose nature I cannot currently discuss. Within this dataset there are about ~4700 unique training images. This dataset is then segmented 90%, 10% train & validation respectively.

Say you trained the above data split on a UNET model with a batch size of 10. You gather up some metrics and boom you're done. (Lets say your Val Loss = X @ 20 epochs) You then decide "let's gather some more data using Keras' data generator class" (i.e. Image Augmentation)

You add +16,000 augmented images to the 4700 images you already have to make a total of ~20,000+ images. You train it on the same UNET model with the same batch size but the validation loss doesn't seem to budge. On top of that it seems a bit higher than the previous model (with only 4000 images) i.e. Val loss = Y > X @ 20 epochs. Moreover, it begins to overfit quite hard around 10-15 epochs. You'd think things would be better right? (law of large numbers!?)

My question is, if you add more data is that a recipe to a better performing model? If not, why not?

My theory is as follows:

1- Maybe the data you're adding doesn't "add" enough "new" information. But if that is true, then won't this apply to most datasets? i.e. What is the difference between a rotated image of a person (using Keras) vs a non augmented image of a person standing straight up? It doesn't really give off new information/features about the person itself. The person is a person whether at 0 deg. or 90 deg. with typical person features. But I figure it might learn new spatial representations or maybe contexts in which you would find a person? Anyone care to elaborate?

(Take this analysis with a grain of salt, I'm spewing ideas)

2- Given a certain problem, either the model itself is not best suited to the task? Or somehow to achieve a validation loss of say 0.1 in this task is notoriously difficult? But then again, difficult why?

Any discussion/answer would be appreciated!

Improve this question
$\endgroup$
2
$\begingroup$

If adding more data means your model fits less well, one invocation of Occam's Razor would be that the initial model wasn't as good as you thought it was, and the new data has revealed this.

Another invocation of Occam's Razor is that you are not actually adding new data. Instead you are generating data based on old data, which isn't strictly theoretically valid for refining models. It's not actually new data, it's just a fiddled repeat of old data and what you saw was a crude estimate of the effect of the fiddling.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Alright, so regarding the first invocation, it fits just as well if just a bit worse (maybe due to its stochastic nature?) As for the second invocation (more interesting), is then what purpose does augmenting images have if it isn't to generate "more new" data for the model, whatever new means here. Besides, one could make the argument that a shift in an original image introduces spatial context into the mix (even though the features of a given class are still the same) no? $\endgroup$ – Kamal Raydan Dec 14 '20 at 19:53
  • $\begingroup$ I come from the world of basic medical research, where anything generated by an algorithm is called "hypothesis generation", if it's even considered. It's simply not data. It can be used to generate a hypothesis for a model that you then confirm by gathering actual data from the outside world. As to why some people use diddled old data as if it were new data, I have no clue. There might be all kinds of mathematical head-standing one could use to rationalize it, but that doesn't turn "old data" + "artificial changes" into "new data. $\endgroup$ – Bryan Dec 14 '20 at 19:57
  • 1
    $\begingroup$ Any "shift" in an original image that is "generated" is just rearranged old information. The "shift" is artificial, hence not a new data collection event. Any information that was not in the original image was intentionally put there and solely represents the algorithm's writers' presumptions. $\endgroup$ – Bryan Dec 14 '20 at 19:58
  • $\begingroup$ Good point. Although I intuitively understand what you are getting at and the reasoning behind it (i.e. That artificially generated data =/= new data) I can't wrap my head around why the model can't extract this "artificially inserted" context in the data. I mean, we might be discussing semantics here but whether new is non artificial or not, should be non-distinguishable by the model since data is data. Or at least I think so. $\endgroup$ – Kamal Raydan Dec 14 '20 at 20:05
  • $\begingroup$ I mean what's stopping me from actually going out and recreating this "artificially augmented" image and calling it "new data". Would it all of a sudden have my model learn "better"? $\endgroup$ – Kamal Raydan Dec 14 '20 at 20:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.