I was running an experiment with neural networks and I could see that training the neural network model with the same input give different results on the test. This should be due to the different weight initializations. The question is how to choose a model from these iterations. Do we need to take the average of results or the best result or the take the average of weights and then retrain and get a single result?
Normally you'd take the best result.
However you run a risk of overfitting on the test set. You might find a set of initial weights that cause the model, when trained, to perform the best on that particular test set, which might not generalize well on different test sets.
I'd suggest having two sets for this purpose, a validation set where you'd perform the initial experiments as you did. Then test the best of these networks on the test set and see if it achieves the same performance as it did on the validation set.