1
$\begingroup$

I am training a simple Neural network with some Dense and Dropout Layers. But on running the fit function, there is no training taking place. My Model is:

import tensorflow as tf
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.models import Sequential

Model = Sequential()

Model.add(Dense(32, input_dim=12, activation='relu'))

for i in range(4):
    Model.add(Dense(2**(5+i), activation='relu'))
    Model.add(Dropout(0.5))

Model.add(Dense(1, activation='softmax'))

Model.summary()
Model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.1),
    loss=tf.keras.losses.BinaryCrossentropy(),
    metrics=[tf.keras.metrics.Accuracy()]
)
my_callbacks = [
    tf.keras.callbacks.EarlyStopping(patience=2),
]
Model.fit(x=X, y=Y, batch_size=32, epochs=20, verbose=1, validation_split=0.1, callbacks=my_callbacks)
Result of Training are:
Epoch 1/20
26/26 [==============================] - 0s 5ms/step - loss: 9.3856 - accuracy: 0.3845 - val_loss: 9.4884 - val_accuracy: 0.3778
Epoch 2/20
26/26 [==============================] - 0s 3ms/step - loss: 9.3856 - accuracy: 0.3845 - val_loss: 9.4884 - val_accuracy: 0.3778
Epoch 3/20
26/26 [==============================] - 0s 3ms/step - loss: 9.3856 - accuracy: 0.3845 - val_loss: 9.4884 - val_accuracy: 0.3778
<tensorflow.python.keras.callbacks.History at 0x7f94a623ffd0>
I tried to see what the Model is predicting:
Model.predict(X)[:10]

Results:

array([[1.],
       [1.],
       [1.],
       [1.],
       [1.],
       [1.],
       [1.],
       [1.],
       [1.],
       [1.]], dtype=float32)

and so on (all predictions are 1.0).

My input dataframes are:

X.head(5)
    0            1           2           3           4           5           6           7           8           9           10          11
0   -0.572351   -0.518084   0.919925    -0.743497   0.743497    -0.50977    -0.32204    0.655011    -0.611972   0.481288    -0.445  -0.503402
1   1.747178    -0.518084   -1.087045   1.344995    -1.344995   1.96167     -0.32204    -1.526692   0.630431    0.481288    -0.445  0.734222
2   -0.572351   -0.518084   0.919925    1.344995    -1.344995   -0.50977    -0.32204    0.655011    -0.301371   -0.479087   -0.445  -0.490356
3   1.747178    -0.518084   -1.087045   1.344995    -1.344995   -0.50977    -0.32204    0.655011    0.397481    0.481288    -0.445  0.382778
4   -0.572351   -0.518084   0.919925    -0.743497   0.743497    -0.50977    -0.32204    0.655011    0.397481    -0.479087   -0.445  -0.487940
Y.head(10):
0    0
1    1
2    1
3    1
4    0
5    0
6    0
7    0
8    1
9    1
Name: Survived, dtype: int64

I am not finding the mistake I am making.

$\endgroup$
2
$\begingroup$

I'm not familiar with the details of Keras specifically, but I see that your output is one node with softmax activation. The $i^{\mathit{th}}$ element of softmax on a preactivation vector $a$ of length $n$ is: \begin{align} \text{softmax}(a)_i & = \frac{\exp(a_i)}{\sum_{i=1}^n \exp(a_i)} \end{align} If $n=1$, we get: \begin{align} \text{softmax}(a)_1 & = \frac{\exp(a_1)}{\sum_{i=1}^1 \exp(a_i)}\\ & = \frac{\exp(a_1)}{\exp(a_1)}\\ & = 1 \end{align} Softmax is useful when you have multiple outputs, not one.

You should be able to just remove the softmax from your output, and specify that the BinaryCrossEntropyLoss is from logits (https://www.tensorflow.org/api_docs/python/tf/keras/losses/BinaryCrossentropy).

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.