I have created my own neural net which is using batch gradient descent. In other words, it trains on batches of examples all at once.
My issue is trying to figure out when to stop the training of the batch. I'll try to make things as understandable as possible since there are so many options, but please comment if you need clarification.
Each batch allows me to compute a list of output errors that correspond to the error the computer has on each example. Each error is a vector with elements equal to the number of output neurons.
Let each error vector be notated by e1, e2, e3, ...
so the list of output errors is [e1, e2, ...]
. Let an error vector in general be notated by e
.
Now there are 2 things we can do to a vector or a list:
- Average: average all of the elements in a vector or list. This will be notated by the function
a()
. For example to find the average error of an example we would doa(e)
- Check: make sure that each element in the vector or list is below a certain value. If any of the elements are too high, we continue to train on the entire batch. (This function can also be applied to a single value.) This will be notated by the function
c()
. For example, if we wanted to make sure the average error ine
was low enough, we would doc(a(e))
Sum: add up all of the elements in the vector or list. This will be notated by the functions()
. For example,s(e)
would be totaling the error found in error vectore
.
Here are the methods I have determined for deciding when to stop gradient descent:
- Iterative: just run the training a certain number of times
- Average of averages: Find the average error of each example, then find the average of those and make sure it's below a certain amount. This is given by the formula
c(a([a(e1), a(e2), ...]))
- Specific error: Check to make sure the error in each output neuron is below a certain amount. This is given by the formula
c(e1), c(e2), ...
- Specific averages: Check to make sure the average error in each example is below a certain amount. This is given by the formula
c([a(e1), a(e2), ...])
Sum of sums: Total up all the error in the entire batch and then make sure it's low enough. This is given by the formulac(s([s(e1), s(e2), ...])
Sum of averages: Find the average error of each example, then add up all those and make sure it's below a certain amount. This is given by the formulac(s([a(e1), a(e2), ...]))
Average of sums: Compute the average total error in each example, and then make sure it's below a certain amount. This is given by the formulac(a([s(e1), s(e2), ...]))
Specific sums: Check to make sure the total of the error in each example is below a certain amount. This is given by the formulac([s(e1), s(e2), ...])
So my question is this:
Which one of these options is most commonly used when training neural nets? Why? (Is there another option that I've missed?)
(Edit: I have found a very general version of this question here, where the answers seem to say there is no specifically good way. If that is still true, please say so in your answer. Note that this is not a duplicate, as I explain the specific methods I would like to use.)
Edit: as A.D pointed out, summing is pretty useless because it depends on how many values are in each list/array/batch. And if you know how many examples there are, you can just use averaging. So, the methods involving "summing" have been cut.