How many times should one train a convolutional neural network? ML engineers usually train 50-100 times a network and take the best model among those. I am wondering how many times a CNN should be trained as training a CNN is costly and time-consuming too. 
 A: If you're looking for an exact number I'm not entirely sure what the answer is to your question, but I believe the error surface complexity of neural networks is determined by the complexity of your network (number of neurons, connectivity and layered structure). It follows that in a highly dense network it is extremely likely to get stuck in a local minimum due to the high gyrification, so retraining is necessary to find the lowest practical point, and more training should be necessary in more highly complex networks. It would be nice if someone formalized a calculus to predict error surface complexity from network complexity and data parameters. Maybe they have but I'm not aware of it.
It is possible to calculate the mean and variance of the correlation of targets and outputs over a number of training sessions, so this could be useful, but higher precision is not always desirable due to overfitting. As a professor of mine once said, working with neural networks is a bit of a dark art.
edit: just wanted to mention that it is possible to iterate over training sessions (including permuted data shuffling both within the training set and between training and test sets) until a certain predictive power that you're happy with is reached. Obviously cost and time are important, but assuming you need a model that predicts to the extent you need it to is the main issue. It should become apparent quite quickly if this process is within your budget. 
