0
$\begingroup$

I try to compare different CNN models. I use Keras and for training, I use a GPU, Google Colab with Tensorflow backend. Unfortunately I'm not able to create the same initial conditions for the CNNs (or in other words: I always get different results). Although by putting the following lines at the top of the code, I get always different results after every run.

from numpy.random import seed
seed(1)
from tensorflow import set_random_seed
set_random_seed(2)

Can it be that it is simply not possible to get reproducible results?

Would it be the best way to simply repeat the training several times and then either calculate a mean (if possible) or simply practice ensemble learning? All without using seeds or random_states or shuffle=False etc.

What would be the best way to compare these models?

Improve this question
$\endgroup$
4
  • $\begingroup$ Not an answer, but one issue is that tf.set_random_seed (tf.random.set_seed in TensorFlow 2) sets the seed for the default graph, but not for individual operations that might have their own random number generators $\endgroup$
    – Artem Mavrin
    Oct 11, 2019 at 21:30
  • $\begingroup$ Oh, that sounds like looking for the needle in a haystack. Actually, I wanted to make this comparison only to determine which model best classifies a given data set $\endgroup$
    – Code Now
    Oct 11, 2019 at 21:46
  • $\begingroup$ You can specify a seed for each operation that involves randomness. This includes obvious things like dropout (both tf.nn.dropout and tf.keras.layers.Dropout accept a seed parameter) as well as more subtle things like the random initialization of weights for dense or convolutional layers (e.g., you can pass a seed parameter to the initializers in the tf.keras.initializers module). $\endgroup$
    – Artem Mavrin
    Oct 11, 2019 at 22:53
  • $\begingroup$ ok, even if I succeed in producing reproducible models, as a beginner in deep learning the question that arises for me is, whether I have really created the best conditions for comparing the performance of different models? Maybe I influence indirectly the so-called randomness in such a way that the result is forced indirectly in a certain direction and the randomness no longer really exists. Wouldn't it be the more reliable way to repeatedly run the models as mentioned above (repeating for example 5 times for each model)? $\endgroup$
    – Code Now
    Oct 11, 2019 at 23:46

1 Answer 1

Reset to default
0
$\begingroup$

From your question it's not clear what you mean with different CNN models. Are you are comparing two models with different architecture, or two instances of the same model with different weights? Since I understand that you are a beginner in deep learning, I'll try to answer for both of the scenarios.

Comparing two different model architectures

In this case the answer is straightforward: compare the average performances of the two models. Run the training several times, introducing diversity in train/validation/test data performing the experiments, for example, in cross validation. You should use the same splits of your dataset for both of the models.

Reproduce results of the same model

This is more tricky. If you are using tensorflow, you should be aware that a "random" operation is in fact ruled by two different seeds: a global seed, set by tf.set_random_seed, and an operation seed, provided as an argument to the operation itself. Once you have set the global seed, the system deterministically picks an operation seed in conjunction with the graph-level seed so that it gets a unique random sequence. However, it's important to set it in the right moment. You probably want to use the same graph and same operations to get the same random numbers in different sessions, so you should set the seed just before running the tf.session.

You can read more about this topic in this discussion

Improve this answer
$\endgroup$
5
  • $\begingroup$ I'm comparing more than 5 models with different architectures and I use the Fashion MNIST data set. How many times do I have to repeat the run for each model? Since the test data set consists of 10000 images, I could do without an additional split in train/validation/test data, right? Would that mean that in this case I don't have to set individual seeds (for reproducibility) for each run? An additional problem is that I have to optimize hyperparameters (via GridSearch), e.g. learning rate, number of epochs, before training. Do I not need seeds at least for tuning? $\endgroup$
    – Code Now
    Oct 18, 2019 at 3:31
  • $\begingroup$ If the models' performances have high variance, you should repeat the runs more; otherwise, you can stick to lower number of iterations. With the experiment set that you have, I'd proceed like this: you can keep fixed the test, but produce N random splits of train/val. You will run GridSearch N times, training on train set and validating on val set. The best model coming from the GridSearch will be tested on the fixed test set. At the end of the day, you will have N test scores to average, for each of your CNN models. The only seed you need to set is the one ruling the train/val split. $\endgroup$
    – arcticriki
    Oct 19, 2019 at 9:00
  • $\begingroup$ You can also think to fix the tf.seed to have similar conditions for the neural networks, but if the models have different architecture I wouldn't care too much about this - differences in architectures are more important than differences in initializations. One thing that for sure could be kept the same for all the models is the parameters grid for the GridSearch. $\endgroup$
    – arcticriki
    Oct 19, 2019 at 9:03
  • $\begingroup$ OK, to sum up your suggestion: run GridSearchCV N times for each model (each of these N runs of GridSearchCV with a different random_state seed for StratifiedKFold). Then testing each of the N best models with the fixed test set and average the results with calculation of the standard deviation, right? And then it is important to note that the different random_state seeds of the N runs of GridSearchCV are the same for each model. $\endgroup$
    – Code Now
    Oct 19, 2019 at 12:19
  • $\begingroup$ Exactly. This is the procedure I'd follow. If you are satisfied with the answer don't forget to vote my answer ;) $\endgroup$
    – arcticriki
    Oct 20, 2019 at 14:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.