2
$\begingroup$

I have the exact same model architecture, one in Keras and one in TensorFlow. The TensorFlow model is actually defined in Keras but uses the TensorFlow session. When I try to fit the data, the TensorFlow version is slow and never gets past 92% training accuracy, whereas the Keras version is lightning fast and reaches 95% in the first epoch. What am I doing wrong here? Is Keras.fit that much better?

Here is the TensorFlow version (with Keras architecture definition layers):

sess = tf.Session()
K.set_session(sess)

input_tensor = tf.placeholder(tf.float32, shape=(None, MAX_SEQUENCE_LENGTH), name="input_tensor")
labels = tf.placeholder(tf.float32, shape=(None, 7))

embedding = Embedding(len(word_index) + 1, embedding_size,
                  weights=[embedding_matrix],
                  input_length=MAX_SEQUENCE_LENGTH,
                  trainable=True)(input_tensor)

# Convolutional
conv = Conv1D(128, 5, activation='relu')(embedding)
conv = Conv1D(64, 5, activation='relu')(conv)
conv = Conv1D(32, 5, activation='relu')(conv)
conv = Conv1D(16, 5, activation='relu')(conv)
flatten = Flatten()(conv)
dense = Dense(128, activation='relu')(flatten)
dropout = Dropout(0.2, name="dropout")(dense)
dense = Dense(Y.shape[1], activation='softmax')(dropout)
output = tf.identity(dense, name="output_tensor")

binary_cross_entropy = tf.reduce_sum(
tf.nn.sigmoid_cross_entropy_with_logits(labels=labels, logits=output), name="xentropy")

train_step = tf.train.AdamOptimizer(1e-4).minimize(binary_cross_entropy)

correct_prediction = tf.equal(tf.argmax(output, 1), tf.argmax(labels, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

builder = tf.saved_model.builder.SavedModelBuilder("./model_keras")

sess.run(tf.global_variables_initializer())

# THIS IS SLOW

with sess.as_default():
    for i in range(200000):
        batch = generate_minibatch_sequences(256)
        if i % 100 == 0:
            train_accuracy = accuracy.eval(feed_dict={
                input_tensor: batch[0], labels: batch[1], K.learning_phase(): 0})
            print("step %d, training accuracy %g"%(i, train_accuracy))
        train_step.run(feed_dict={input_tensor: batch[0],
                              labels: batch[1], K.learning_phase(): 1})

Now the exact same architecture using Keras.fit is orders of magnitude faster and converges right away:

inputs = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
embedding = Embedding(len(word_index) + 1, embedding_size,
                weights=[embedding_matrix],
                input_length=MAX_SEQUENCE_LENGTH,
                trainable=True)(inputs)
conv = Conv1D(128, 5, activation='relu')(embedding)
conv = Conv1D(64, 5, activation='relu')(conv)
conv = Conv1D(32, 5, activation='relu')(conv)
conv = Conv1D(16, 5, activation='relu')(conv)
flatten = Flatten()(conv)
dense = Dense(128, activation='relu')(flatten)
dropout = Dropout(0.2, name="dropout")(dense)
dense = Dense(Y.shape[1], activation='softmax')(dropout)
model = Model(inputs, dense)
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])
print(model.summary())

# THIS IS FAST

history = model.fit(x_train, y_train,
                validation_data=(x_val, y_val),
                nb_epoch=50,
                batch_size=128,
                callbacks=[ModelCheckpoint(
                    "toxic-comments-lstm.{epoch:02d}-{val_acc:.4f}.hdf5")])

Why would fit be so much better out of the box? Do I need to implement whatever fancy magic fit does under the hood?

$\endgroup$

0