3
$\begingroup$

I tried out a simple GRU network with only 1 layer, 1 input tensor and 1 output, to verify its actual network connection (input nodes->hidden layer->output) by doing the manual calculation with the actual GRU equations.

The blogs/website/videos which I gone thru to understand about the math behind GRU networks:

  1. Youtube: Actual matrix representation of a neural network.
  2. Blog: The tensorflow API: GetWeights() return the weights and biases for update gate, reset gate and hidden state, in this order: GRU.get_weights() = [[W_z; W_r; W_h], [U_z; U_r; U_h], [bias_z; bias_r; bias_h]]
  3. GRU python code: An example of GRU implementation with TensorFlow Keras API.
  4. Wiki: GRU equations

enter image description here

However, my calculation does not match with the network prediction output. The codes which I tried (modified from GRU python code) as below:

#Step 1: Import Libraries
import keras
from keras.models import Sequential
from keras.layers import GRU
import numpy as np

#Step 2: Define the model
model = Sequential()
model.add(GRU(1, input_shape=(4,1)))
model.compile(optimizer='adam',loss='mse')


#Step 3: Define a sample array to run the model
#input time steps
y = np.array([[1, 2],[9, 8]]).reshape((1,4,1))

#make and show prediction
print(model.predict(y))


#To get model weight
#ref: https://dieuwkehupkes.nl/finding-recurrent-network-weights-in-keras-models/
#based on the link above, the return weights are in the order of:[[W_z; W_r; W_h], [U_z; U_r; U_h], [bias_z; bias_r; bias_h]]
GRU_layer = model.layers[0]
weights = GRU_layer.get_weights()
print(weights)

Outputs:

[[0.9680257]]  -->This is network prediction output

Weight = [
array([
[-0.90954137, 1.0425225 ,
-0.04404902]], dtype=float32),

array([[0.24168766, 0.8981342 , 0.36734456]], dtype=float32),

array([[0., 0., 0.], [0., 0., 0.]], dtype=float32)
]

My calculation based on GRU equations (with the weights retrieved thru get_weights()):

Z  =  σ(-0.90954137·[1,2,9,8] + 0.24168766(0) + 0)
   = σ(-18.1908274+0+0)
   = 1/(1+e^(-((-18.1908274)))
   = 1/((79,465,014.420602057659182852894637)
   = 1.2584154263246953218654856950477e-8

r= σ(1.0425225· [1,2,9,8] + 0.8981342(0) +0)
  = σ(20.85045)
  = 1/(1+e^(-(20.85045))
  = 1/(1.00000000088)
  = 0.99999999912

h ̃=  tanh⁡〖(-0.04404902·[1,2,9,8]+0.99999999912∗(0.36734456〗)·0+0)
          = tanh(-0.8809804)
          = -0.70691013301376

h=0∗0+(1-0)∗(-0.70691013301376)=-0.70691013301376 (not matched with network prediction value: 0.9680257)

So, I would like to get some advice on how the matrix calculation being done for the GRU network, (which can correlate to nodes connection (from input to output) in the GRU architecture) to get the final prediction value by TensorFlow Keras GRU API. Thanks.

$\endgroup$
2
  • 1
    $\begingroup$ pretty sure that the problem is that [1,2,9,8] is interpreted as the sequence, therefore to the GRU is first fed the 1, then the 2, then the 9 and finally the 8 $\endgroup$
    – Alberto
    Commented Jul 10, 2022 at 10:24
  • 1
    $\begingroup$ and btw, -0.90954137·[1,2,9,8] is definitely not -18.1908274... a scalar times a vector is a vector $\endgroup$
    – Alberto
    Commented Jul 10, 2022 at 10:25

1 Answer 1

1
$\begingroup$

The tensorflow.keras.layers.GRU uses the following formula to calculate the new state h = z * h_old + (1 - z) * hnew ,which is based on "Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation" by Kyunghyun Cho et al.

On the other hand the formula in wiki but also on "Neural machine translation by jointly learning to align and translate" by Dzmitry Bahdanau et al. is as follows: h = (1 - z) * h_old + z * hnew

The difference in the order of the terms does not affect the performance of the GRU cell, since the sigmoid activation function used to compute the update gate, z, produces a value between 0 and 1, making both formulas symmetric.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.