# How to compare CNN models with non-reproducible results?

I try to compare different CNN models. I use Keras and for training, I use a GPU, Google Colab with Tensorflow backend. Unfortunately I'm not able to create the same initial conditions for the CNNs (or in other words: I always get different results). Although by putting the following lines at the top of the code, I get always different results after every run.

from numpy.random import seed
seed(1)
from tensorflow import set_random_seed
set_random_seed(2)


Can it be that it is simply not possible to get reproducible results?

Would it be the best way to simply repeat the training several times and then either calculate a mean (if possible) or simply practice ensemble learning? All without using seeds or random_states or shuffle=False etc.

What would be the best way to compare these models?

• Not an answer, but one issue is that tf.set_random_seed (tf.random.set_seed in TensorFlow 2) sets the seed for the default graph, but not for individual operations that might have their own random number generators Oct 11, 2019 at 21:30
• Oh, that sounds like looking for the needle in a haystack. Actually, I wanted to make this comparison only to determine which model best classifies a given data set Oct 11, 2019 at 21:46
• You can specify a seed for each operation that involves randomness. This includes obvious things like dropout (both tf.nn.dropout and tf.keras.layers.Dropout accept a seed parameter) as well as more subtle things like the random initialization of weights for dense or convolutional layers (e.g., you can pass a seed parameter to the initializers in the tf.keras.initializers module). Oct 11, 2019 at 22:53
• ok, even if I succeed in producing reproducible models, as a beginner in deep learning the question that arises for me is, whether I have really created the best conditions for comparing the performance of different models? Maybe I influence indirectly the so-called randomness in such a way that the result is forced indirectly in a certain direction and the randomness no longer really exists. Wouldn't it be the more reliable way to repeatedly run the models as mentioned above (repeating for example 5 times for each model)? Oct 11, 2019 at 23:46

From your question it's not clear what you mean with different CNN models. Are you are comparing two models with different architecture, or two instances of the same model with different weights? Since I understand that you are a beginner in deep learning, I'll try to answer for both of the scenarios.

Comparing two different model architectures

In this case the answer is straightforward: compare the average performances of the two models. Run the training several times, introducing diversity in train/validation/test data performing the experiments, for example, in cross validation. You should use the same splits of your dataset for both of the models.

Reproduce results of the same model

This is more tricky. If you are using tensorflow, you should be aware that a "random" operation is in fact ruled by two different seeds: a global seed, set by tf.set_random_seed, and an operation seed, provided as an argument to the operation itself. Once you have set the global seed, the system deterministically picks an operation seed in conjunction with the graph-level seed so that it gets a unique random sequence. However, it's important to set it in the right moment. You probably want to use the same graph and same operations to get the same random numbers in different sessions, so you should set the seed just before running the tf.session.