1
$\begingroup$

I have model A:

model = Sequential()
tuple_kernel_size = (3, 3)
pool_size = (2, 2)

model.add(Conv2D(32, tuple_kernel_size, activation='relu', input_shape=(input_dim, input_dim, 3)))
model.add(MaxPooling2D(pool_size))
model.add(Conv2D(64, tuple_kernel_size))
model.add(MaxPooling2D(pool_size))
model.add(Flatten())
model.add(Dense(units=64, activation='relu'))
model.add(Dropout(0.4))
model.add(Dense(units=32, activation='relu'))
model.add(Dropout(0.4))
model.add(Dense(units=1, activation='sigmoid'))

model.compile(loss='binary_crossentropy', metrics=['acc'])

As you can see the second Conv2D layer does not have an activation function set. This is the performance of that model during training. As you can see the accuracy is mostly stuck around 0.5 (is it a binary classification problem with balanced classes).

enter image description here

However when I add the relu activation function to the second conv layer and leave the rest the same. The model trains much much better, what is going on here? Could someone provide an intuitive understanding of why adding an activation function here matters that much for performance. The loss also keeps dropping while it is stuck around 0.7 for the model without the activation function. Thanks!! Model B Performance

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Not using an activation function means that the output of the convolution layer's output is a (masked) linear combination of the input. Linear networks are not very expressive, so it's not surprising that this model does not fit the data very well. The downsides to linear networks are explained in stats.stackexchange.com/questions/228296/…. $\endgroup$
    – Sycorax ♦
    Commented Jun 14, 2022 at 14:36

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.