Why does a neural network trained with random data and fixed initialization have different weights between runs? I wrote a simple code that creates a neural network with two dense layers and then trains it. In this code, the initial coefficients of each layer are fixed. Why do the answers change every time? (In situations where layers such as dropout are not used and the initial coefficients are also fixed)
```
import numpy as np
import keras
from keras import layers

x = np.random.random([200,4])
y = np.random.random([200,1])

model = keras.Sequential()
model.add(layers.Dense(units=4,activation='relu',input_shape (x.shape[1:]),kernel_initializer='ones',bias_initializer='ones'))
model.add(layers.Dense(units=1,kernel_initializer='ones',bias_initializer='ones'))
opt = keras.optimizers.Adam(learning_rate = 0.0001)
model.compile(loss = keras.losses.categorical_crossentropy, optimizer= opt, metrics=['mae'])

history = model.fit(x, y, epochs=50)
```

 A: Your input and output are random. Every time you run the code, they take different values; of course the neural network gives different results for the different data.
This turns out not to be a statistics problem. Just stick np.random.seed(2022) (or some other number, but I learned to go with the year from our late friend, BruceET) at the top of your code to lock in your $X$ and $y$ variables. The generation of those variables will be random, but it will be the same random generation every time you run the code.
A: If you are using Tensorflow as the backend for Keras, there are a couple of other reasons why your model can learn different weights.

*

*Tensorflow has its own random number generators, so you also need to set the seeds for these. In addition to the numpy seed, I also set these seeds:

        os.environ['PYTHONHASHSEED'] = str(seed)
        random.seed(seed)
        tf.random.set_seed(seed)



*If you are using GPUs, there is some inherent randomness in the way sequencing of some operations happens. I've tried following various suggestions I've found from online searches (sorry I don't have references for these) but the only way I've found to get completely reproducible models using Keras/Tensorflow is to disable GPU processing.

EDIT:
One reason for setting the TF random seed, even though the weight initialisation is to constant values, is that shuffling (the training data before each epoch) which is enabled by default is a random operation.
This link to a Keras FAQ about reproducibility provides some more information about why each of the random seeds needs to be set. I noticed this FAQ recommends (a) setting the PYTHONHASHSEED environment variable before running the Python program and (b) setting it to 0 (which according to the Python documentation disables hash randomisation) rather than a seed. While I found setting it to a fixed seed within the program sufficient, this documentation states: "First, you need to set the PYTHONHASHSEED environment variable to 0 before the program starts (not within the program itself). This is necessary in Python 3.2.3 onwards to have reproducible behavior for certain hash-based operations (e.g., the item order in a set or a dict, see Python's documentation or issue #2280 for further details)."
This code snippet is also from the above FAQ reference and explains why each seed needs to be set:
import numpy as np
import tensorflow as tf
import random as python_random

# The below is necessary for starting Numpy generated random numbers
# in a well-defined initial state.
np.random.seed(123)

# The below is necessary for starting core Python generated random numbers
# in a well-defined state.
python_random.seed(123)

# The below set_seed() will make random number generation
# in the TensorFlow backend have a well-defined initial state.
# For further details, see:
# https://www.tensorflow.org/api_docs/python/tf/random/set_seed
tf.random.set_seed(1234)

And for anyone trying to get reproducible results when using a GPU, these two GitHub NVIDIA references may be useful: https://github.com/NVIDIA/framework-determinism/blob/master/doc/tensorflow_status.md and https://github.com/NVIDIA/framework-determinism/blob/master/doc/tensorflow.md
