I have a very broad question about the general procedure of training a NN. I am not too concerned about the precise algorithms in this question at the moment. But there is one thing bothering me.
Suppose I have a set of samples that are fed into a NN, either as a batch or online, and I also have corresponding ideal(s) for each batch that I want to compare the output of the NN to. Do I carry out epoch iterations on a single batch basis and consider that to be a single NN? Does that mean that at the end of the process I end up with p NNs corresponding to the p number of batches?
1. What is the process of combining a set of p NNs derived from p training batches into a single NN? The implicit question here is that if I train a NN with batch/sample p, will the NN continue to take all the previous training (ie. batch samples 0 to (p - 1)) into account? I imagine that as you progress towards the end of the data set, the NN will be such that it takes account of only the most recent sample data of the entire set of samples that it has been fed. Is this right?
2. In the nomenclature of batch training, say if we want to combine the results from a series of NNs, then I presume the NNs should be averaged together - namely all the weights averaged. Or perhaps they should be summed. I don't quite understand why the weights in a typical batch scenario are summed and not averaged?