NNs: Multiple Sigmoid + Binary Cross Entropy giving better results than Softmax + Categorical Cross Entropy

Question

I am experimenting with classifying documents into one (and only one) of 20 classes ( 20_newsgroups dataset) using Keras.

I'm using standard TF-IDF features and .2 validation split for this setting.

It's a 20-way classification task, so I thought it was better to go with a softmax at the output and a 'categorical_crossentropy' loss function, giving me 85% validation accuracy:

inputs = Input(shape=(19860,))

x = Dense(64,activation='relu')(inputs)
x = Dropout(0.5)(x)
x = Dense(64,activation='sigmoid')(x)
x = Dropout(0.5)(x)
preds = Dense(20,activation='softmax')(x)

model = Model(inputs=inputs,outputs=preds)
model.compile(loss='categorical_crossentropy',
             optimizer='adam',
             metrics=['acc'])
model.fit(X_train,Y_train, validation_data=(X_val, Y_val),
     epochs=10, batch_size=32)

Results (after 10 epochs): loss: 0.4056 - acc: 0.8792 - val_loss: 0.4842 - val_acc: 0.8542

Thing is, I get better results if I train the same model using 20 individual sigmoid units at the output layer and a 'binary_crossentropy' loss function! It gives me a whopping 99% validation accuracy:

inputs = Input(shape=(19860,))

x = Dense(64,activation='relu')(inputs)
x = Dropout(0.5)(x)
x = Dense(64,activation='sigmoid')(x)
x = Dropout(0.5)(x)
preds = Dense(20,activation='sigmoid')(x)

model = Model(inputs=inputs,outputs=preds)
model.compile(loss='binary_crossentropy',
             optimizer='adam',
             metrics=['acc'])
model.fit(X_train,Y_train, validation_data=(X_val, Y_val),
     epochs=10, batch_size=32)

Results (after 10 epochs): loss: 0.0510 - acc: 0.9819 - val_loss: 0.0443 - val_acc: 0.9851

If I understand things correctly, the multiple sigmoid setup assumes label independence, which is clearly not the case since there can be only one label for each document.

So how can I get a better results using individual, independent sigmoid output units rather than a single softmax output unit?

Could anyone give me some intuition as to what's maybe happening here?

Answer 1

For your problem, the good metric is the categorical_accuracy. What happens is that when you set the loss to be binary_crossentropy and metrics to accuracy then keras assumes that the good metric is binary_accuracy which is just plain wrong when there is more than 2 labels.

What you have to do is to specify explicitly that the metric is categorical_accuracy like this:

from keras.metrics import categorical_accuracy
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[categorical_accuracy])

see the details in this answer: https://stackoverflow.com/a/46038271/6338493