How is it possible? Training and validation loss curves were decreasing while training data size was increasing I'm really puzzled... I’ve learned and observed that training loss / error increases with training data size as stated in Dr Andrew Ng’s ML course.

I’ve recently experienced an anomaly. Training loss and validation loss curves were decreasing while training data size was increasing. The code was plotting after all the training was entirely done. In fact, I’ve trained 10 different DNN models with increasing quantity of data.

I suspect that Dr Ng’s drawing assumes that the number of data points is already pretty large. The drawing does not show the training error curve for small training sets. Maybe in that area, for high capacity DNN both training loss and validation loss can decrease together as the training data size increases. Maybe, maybe not... I don't really know... 
That said, I’ve got the Dr Ng’s described behavior with small classical machine learning models using the same code but different data for both experiments, I just have to replace the model.

 A: I think your example behaves differently due to regularization.
In the case of a very small dataset, an unregularized model should be able to "memorize" the training set and thus reach zero training error. This is what Andrew Ng talks about.
However, if you add some regularization to your model (such as weight decay), especially in the cases of very small dataset, the regularizer will prevent the model to overfit and instead it will push its parameters towards the regularized solution. In a well regularized model, the training and the validation curve should behave similarly.

Update:
If you encounter this in an unregularized network, there may be two further problems:


*

*There is a bug in your code. Reaching zero training error on a tiny dataset (or even a single element) is a good sanity check. See What should I do when my neural network doesn't learn?

*Your data is inconsistent: There are multiple samples with the same value but different label. You can never reach zero training error on such data.
