# Training a deep neural network using a changing (increasing) dataset

I am trying to train a deep autoencoder (applies also for other architectures) in the following way:

Step 1) I start with a fix dataset of e.g. 10k samples.

Step 2) A training "loop" consists of 1000 epochs going through the whole dataset with mini-batches of size e.g. 64.

Step 3) After one "loop" the trained AE is used to generate more data, which (after some processing) is added to the dataset.

After, steps 2-3 are repeated for a predefined number of loops.

Question: What would be the best way to choose the number of iterations for one epoch?

a) Calculate it as: num_iter = len(dataset)/batch_size, which increases everytime new data is added.

b) Keep it constant e.g. num_iter = 200

• Seems like a would be best if I understand. You want to hit every training example available. But I'd be more concerned that the new data you generate doesn't match the distribution of whatever data your network will see when it comes time to use it and it will actually make it perform worse. Make sure to try it with and without adding extra data as it trains to see. – Frobot Jun 6 '19 at 4:26
• The data comes from a [0,1] range and in the first step it is drawn uniformly. In the further set the data is a subset of that space, so I guess it shouldn't be to far off from the initial distribution. I've been using a) so far, and what it does is that after the first loop the loss already goes down significantly, and adding new data and subsequent loops just keep the loss around that value. I'm not sure if this can be improved in any way.. – El Rakone Jun 6 '19 at 11:15