0
$\begingroup$

I'm new to CNN and deep learning in general, forgive my questions if sound too trivial.

I'm trying replicate results from a paper on transportation mode detection using convolutional neural network, starting with the very show network. I trained the model 3 times (using same code and data from the paper), but I'm getting slightly varying results from my training, none of which matches the paper's accuracy exactly. For instance, the models' accuracy for the 3 experiments I run give: 75.57% 75.85% and 76.39%, and the paper's accuracy for this model is 74.5%.

Questions:

  1. Shouldn't the models' accuracy be the same in all cases (at least for the experiments I repeated)?
  2. Could this be affected my kernel/filter initialization?
  3. I read from some resources, that you randomly initialize the filter, but I cannot figure out where exactly in the code, is the filter actually initialized.

Results from the experiments I run:

Exp1 (accuracy = 0.7557)
+---+-----------+--------+----------+---------+
|   | precision | recall | f1-score | support |
+---+-----------+--------+----------+---------+
| 0 | 0.761     | 0.932  | 0.838    |    2122 |
| 1 | 0.835     | 0.724  | 0.776    |    1121 |
| 2 | 0.724     | 0.694  | 0.709    |    1439 |
| 3 | 0.64      | 0.495  | 0.558    |     888 |
| 4 | 0.796     | 0.734  | 0.764    |     918 |
+---+-----------+--------+----------+---------+

Exp2 (accuracy = 0.75848)
+---+-----------+--------+----------+---------+
|   | precision | recall | f1-score | support |
+---+-----------+--------+----------+---------+
| 0 | 0.773     | 0.927  | 0.843    |    2122 |
| 1 | 0.859     | 0.725  | 0.787    |    1121 |
| 2 | 0.73      | 0.717  | 0.723    |    1439 |
| 3 | 0.625     | 0.481  | 0.544    |     888 |
| 4 | 0.796     | 0.734  | 0.764    |     918 |
+---+-----------+--------+----------+---------+

Exp3 (accuracy = 0.76387)
+---+-----------+--------+----------+---------+
|   | precision | recall | f1-score | support |
+---+-----------+--------+----------+---------+
| 0 | 0.777     | 0.931  | 0.847    |    2122 |
| 1 | 0.848     | 0.727  | 0.783    |    1121 |
| 2 | 0.719     | 0.728  | 0.723    |    1439 |
| 3 | 0.668     | 0.495  | 0.569    |     888 |
| 4 | 0.78      | 0.74   | 0.76     |     918 |
+---+-----------+--------+----------+---------+

Source code (can't spot filter initialization step):

import numpy as np
import keras
from keras.models import Sequential
from keras.layers import Dense, Conv2D, Flatten
import pickle
from keras.optimizers import Adam
import random
from sklearn.model_selection import train_test_split
import time
import tensorflow as tf
from sklearn.metrics import classification_report
from keras.callbacks import ModelCheckpoint
tf.Session(config=tf.ConfigProto(allow_soft_placement=True, log_device_placement=True))
sess = print(tf.Session(config=tf.ConfigProto(log_device_placement=True)))

start_time = time.clock()
np.random.seed(7)
random.seed(7)

filename = 'Revised_KerasData_Smoothing.pickle'
with open(filename, mode='rb') as f:
    TotalInput, FinalLabel = pickle.load(f, encoding='latin1')  # Also can use the encoding 'iso-8859-1'

NoClass = len(list(set(np.ndarray.flatten(FinalLabel))))
Threshold = len(TotalInput[0, 0, :, 0])

# Making training and test data: 80% Training, 20% Test
Train_X, Test_X, Train_Y, Test_Y_ori = train_test_split(TotalInput, FinalLabel, test_size=0.20, random_state=7)

Train_Y = keras.utils.to_categorical(Train_Y, num_classes=NoClass)
Test_Y = keras.utils.to_categorical(Test_Y_ori, num_classes=NoClass)

# Model and Compile
model = Sequential()
activ = 'relu'
model.add(Conv2D(32, (1, 3), strides=(1, 1), padding='same', activation=activ, input_shape=(1, Threshold, 4)))
model.add(Conv2D(32, (1, 3), strides=(1, 1), padding='same', activation=activ))

model.add(Flatten())
model.add(Dense(NoClass, activation='softmax'))

optimizer = Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)
model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])
# checkpoint
filepath="CNN-A-weights.best.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]
offline_history = model.fit(Train_X, Train_Y, epochs=62, batch_size=64, shuffle=False,
                            validation_data=(Test_X, Test_Y), callbacks=callbacks_list)
hist = offline_history
A = np.argmax(hist.history['val_acc'])
print('the optimal epoch size: {}, the value of high accuracy {}'.format(hist.epoch[A], np.max(hist.history['val_acc'])))


# Calculating the test accuracy, precision, recall
# Prediction
# load weights
model.load_weights("CNN-A-weights.best.hdf5")
# Compile model (required to make predictions)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
Pred = model.predict(Test_X, batch_size=32)
Pred_Label = np.argmax(Pred, axis=1)

# Precision, recall, F-score
print(classification_report(Test_Y_ori, Pred_Label, digits=3))
$\endgroup$
1
$\begingroup$

Your CNN is being trained using Adam, which is an Adaptive stochastic optimization method similar stochastic gradient decent.

The key word here is stochastic (random). Instead of calculating weight updates on the whole dataset, random mini-batches are used. This means that every time you train your network, you will learn a different set of weights.

Different weights means different predictions, thus the slight changes in accuracy you are observing.

$\endgroup$
2
  • $\begingroup$ Thank you for your detailed answer. $\endgroup$
    – arilwan
    Oct 3 '19 at 11:45
  • $\begingroup$ Just curious to know more, can this behaviour be controlled e.g. using some sort of "seed" variable? $\endgroup$
    – arilwan
    Oct 3 '19 at 14:31

Not the answer you're looking for? Browse other questions tagged or ask your own question.