1
$\begingroup$

I am experimenting with classifying documents into one (and only one) of 20 classes ( 20_newsgroups dataset) using Keras.

I'm using standard TF-IDF features and .2 validation split for this setting.

It's a 20-way classification task, so I thought it was better to go with a softmax at the output and a 'categorical_crossentropy' loss function, giving me 85% validation accuracy:

inputs = Input(shape=(19860,))

x = Dense(64,activation='relu')(inputs)
x = Dropout(0.5)(x)
x = Dense(64,activation='sigmoid')(x)
x = Dropout(0.5)(x)
preds = Dense(20,activation='softmax')(x)

model = Model(inputs=inputs,outputs=preds)
model.compile(loss='categorical_crossentropy',
             optimizer='adam',
             metrics=['acc'])
model.fit(X_train,Y_train, validation_data=(X_val, Y_val),
     epochs=10, batch_size=32)

Results (after 10 epochs): loss: 0.4056 - acc: 0.8792 - val_loss: 0.4842 - val_acc: 0.8542

Thing is, I get better results if I train the same model using 20 individual sigmoid units at the output layer and a 'binary_crossentropy' loss function! It gives me a whopping 99% validation accuracy:

inputs = Input(shape=(19860,))

x = Dense(64,activation='relu')(inputs)
x = Dropout(0.5)(x)
x = Dense(64,activation='sigmoid')(x)
x = Dropout(0.5)(x)
preds = Dense(20,activation='sigmoid')(x)

model = Model(inputs=inputs,outputs=preds)
model.compile(loss='binary_crossentropy',
             optimizer='adam',
             metrics=['acc'])
model.fit(X_train,Y_train, validation_data=(X_val, Y_val),
     epochs=10, batch_size=32)

Results (after 10 epochs): loss: 0.0510 - acc: 0.9819 - val_loss: 0.0443 - val_acc: 0.9851

If I understand things correctly, the multiple sigmoid setup assumes label independence, which is clearly not the case since there can be only one label for each document.

So how can I get a better results using individual, independent sigmoid output units rather than a single softmax output unit?

Could anyone give me some intuition as to what's maybe happening here?

$\endgroup$
2
$\begingroup$

For your problem, the good metric is the categorical_accuracy. What happens is that when you set the loss to be binary_crossentropy and metrics to accuracy then keras assumes that the good metric is binary_accuracy which is just plain wrong when there is more than 2 labels.

What you have to do is to specify explicitly that the metric is categorical_accuracy like this:

from keras.metrics import categorical_accuracy
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[categorical_accuracy])

see the details in this answer: https://stackoverflow.com/a/46038271/6338493

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.