1
$\begingroup$

I have a dataset with 10 input categorical features and one output categorical feature with class 0 and 1. X_train follows a 3D array so I have done label encoding beforehand on the dataset. I have applied categorical_crossentropy but I am getting 26% accuracy with activation function sigmoid. When I apply binary_crossentropy, the accuracy drastically increased to 98%.

model = Sequential()
model.add(LSTM(256, input_shape=(n_timesteps,n_features),recurrent_activation='hard_sigmoid'))
model.add(Dense(16))
model.add(Dense(n_outputs, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

and dataset is divided as:

X_train:  (430000, 5, 10)
y_train:  (430000, 1)
$\endgroup$
2
  • $\begingroup$ What is the value of n_outputs? Note that there are circumstances when the two losses are equivalent, but it's not clear that those circumstances exist in your code; see stats.stackexchange.com/q/260505/22311 $\endgroup$
    – Sycorax
    Feb 27 at 14:19
  • $\begingroup$ n_outputs is 1 and the model is predicting only class 0, not 1. Please suggest how can I improve. $\endgroup$
    – be_real
    Feb 28 at 9:33

1 Answer 1

0
$\begingroup$

There are circumstances when the two losses are equivalent, but those circumstances do not exist in OP's code.

In a comment, OP writes that they only have one output neuron.

With 1 output neuron and binary cross-entropy, the model outputs a single value $p$ abd loss for one example is computed as

$$ L_b = -y \log p - (1 - y) \log (1 - p), $$ which is the correct way to compute the loss.

However, with 1 output neuron and categorical cross-entropy, the loss is computed as

$$ L_c = -y \log p $$

which is clearly different because it fixes $(1-y) \log(1-p)=0$. This loss is obviously bogus because it is minimized at $L_c = 0$ by setting $p=1$ regardless of the input, resulting in a totally useless model.

To use categorical cross-entropy correctly, OP needs to make these changes

  • use $k$ output neurons (one for each of the $k$ classes). In OP's particular case, $k=2$
  • these output neurons need to be a probability vector: the neurons sum to 1 for all inputs, and all values are non-negative. The standard way to do this is to use a softmax activation in the output layer.

After making these changes, the loss will be computed correctly when using categorical cross-entropy. This is because what we want to have is the model outputs $p_1, p_2$ so that the loss is

$$ L_c = -y \log p_1 - (1 - y) \log p_2 $$

where $ 0 \le p_i \le 1$ and $p_1 + p_2 = 1$. In this setting, it's simple algebra to show that $L_c = L_b$, as desired.

$\endgroup$
1
  • $\begingroup$ thank you for the explanation!! $\endgroup$
    – be_real
    Mar 1 at 14:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.