General process of training a Neural Network [closed]

Question

I have a very broad question about the general procedure of training a NN. I am not too concerned about the precise algorithms in this question at the moment. But there is one thing bothering me.

Suppose I have a set of samples that are fed into a NN, either as a batch or online, and I also have corresponding ideal(s) for each batch that I want to compare the output of the NN to. Do I carry out epoch iterations on a single batch basis and consider that to be a single NN? Does that mean that at the end of the process I end up with p NNs corresponding to the p number of batches?

1. What is the process of combining a set of p NNs derived from p training batches into a single NN? The implicit question here is that if I train a NN with batch/sample p, will the NN continue to take all the previous training (ie. batch samples 0 to (p - 1)) into account? I imagine that as you progress towards the end of the data set, the NN will be such that it takes account of only the most recent sample data of the entire set of samples that it has been fed. Is this right?

2. In the nomenclature of batch training, say if we want to combine the results from a series of NNs, then I presume the NNs should be averaged together - namely all the weights averaged. Or perhaps they should be summed. I don't quite understand why the weights in a typical batch scenario are summed and not averaged?

Answer 1

Good question!

A Clarification

In your question, you talk about training multiple NNs with different batches. Typically you are trying to train a single NN. After training on a single batch, you update the NN so that its parameters are slightly different when it is trained on the next batch. One situation in which you would train different NNs on different batches and then combine them together later is when you're doing distributed training. This blog gives a good introduction to how to combine neural nets trained on different subsets of your data. For the rest of this answer, I'll assume we're talking about training a single NN in the non-distributed scenario.

I understand your first question correctly, you're asking: When you train a neural net (NN) on a lot of data, does it learn to perform well on all the data or just the data i has seen most recently?

Shuffling Helps

If the order data is fed into the NN has a certain pattern, your model might only learn to perform well on the data it has seen recently (for instance, if you're training a NN on a cat photo classification problem and you feed it all of the cat pictures followed by all the non-cat pics, the NN might learn to just predict "cat" at the start of training and "non-cat" at the end. However, if each batch contains a mixture of cat and non-cat photos, your NN will have to learn more generalizable methods of predicting the label of an image.

There might be some instances when you're doing online training where you WANT your model to learn more from new experience than from older training examples if you think that new patterns may emerge in data over time. In that case, it's fine to train your model on each new batch of data as you obtain it without shuffling.

Here are some other posts on shuffling.

Question 2

Again, I'd like to reiterate that we are only training a single neural net here. Disributed training is typically the only case where you might average NN weights.

While you're training on a single machine, the typical thing which is summed or averaged is the loss from each training example in your batch. Both methods work, but averaging is often preferred since it makes it easier to compare models trained with different batch sizes. Some frameworks give you the option to choose (such as pytorch).