What is the reason that reduce training time over epoch for LSTM?

Question

I am training and recurrent neural network and observed less time is needed over time.

What could be the reason? I would think calculating the gradient, and update the parameters in the network would be using some "constant time" for a given machine (even reach the local minima). Why after warning up, the time needed is reduced?

---

Using TensorFlow backend.
Train on 8000 samples, validate on 2000 samples
Epoch 1/20
8000/8000 [==============================] - 32s 4ms/step - loss: 2.0374 
Epoch 2/20
8000/8000 [==============================] - 19s 2ms/step - loss: 1.8993 
Epoch 3/20
8000/8000 [==============================] - 19s 2ms/step - loss: 1.7993 
Epoch 4/20
8000/8000 [==============================] - 19s 2ms/step - loss: 1.7140 
Epoch 5/20
8000/8000 [==============================] - 19s 2ms/step - loss: 1.6368 
Epoch 6/20
8000/8000 [==============================] - 19s 2ms/step - loss: 1.5538 
Epoch 7/20
8000/8000 [==============================] - 19s 2ms/step - loss: 1.4761 
Epoch 8/20
8000/8000 [==============================] - 19s 2ms/step - loss: 1.4070 
Epoch 9/20
8000/8000 [==============================] - 18s 2ms/step - loss: 1.3496 
Epoch 10/20
8000/8000 [==============================] - 19s 2ms/step - loss: 1.2979 
Epoch 11/20
8000/8000 [==============================] - 16s 2ms/step - loss: 1.2531 
....

Answer 1

There are many possible reasons for this:

1. In some NN frameworks, the first step of training can take considerable time as the entire computation graph is optimized (constant folding, subexpr elimination, etc).
2. Depending on the system, after you read a data file, it may be stored in a disk cache or in page cache on memory, which greatly accelerates subsequent reads.
3. If you are training on the same GPU which is hooked up to the display, that can interfere with training speed.

As far as I can tell, only the first epoch was slow for you -- the fluctuations in 9 and 11 can probably be ignored.