2
$\begingroup$

In Tensorflow, how do you implement the LeNet-5 pooling layers with trainable coefficient and bias terms?

Reading through the LeNet-5 paper, the subsample layers are described as follows:

Layer S2 is a sub-sampling layer with 6 feature maps of size 14x14. Each unit in each feature map is connected to a 2x2 neighborhood in the corresponding feature map in C1. The fout inputs to a unit in S2 are added, then multiplied by a trainable coefficient, and added to a trainable bias. The result is passed through a sigmoidal function. The 2x2 receptive fields are non-overlapping, therefore feature maps in S2 have half the number of rows and columns as feature maps in C1. Layer S2 has 12 trainable parameters and 5,880 connections.

http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf

However, in my search for examples of implementing LeNet-5 in Tensorflow, I haven't seen this pooling layer implemented with the trainable coefficient and bias. Instead, something like the following is used:

model = keras.Sequential()
model.add(layers.Conv2D(filters=6, 
                        kernel_size=(5, 5), 
                        activation='tanh', 
                        input_shape=(28,28,1),
                        padding='same'))
model.add(layers.AveragePooling2D(pool_size=(2, 2), 
                                  strides=(2, 2), 
                                  padding='valid'))
model.add(layers.Conv2D(filters=16, 
                        kernel_size=(5, 5), 
                        activation='tanh',
                        padding='valid'))
model.add(layers.AveragePooling2D(pool_size=(2, 2), 
                                  strides=(2, 2), 
                                  padding='valid'))
model.add(layers.Flatten())
model.add(layers.Dense(units=120, activation='tanh'))
model.add(layers.Dense(units=84, activation='tanh'))
model.add(layers.Dense(units=10, activation = 'softmax'))

Calling model.summary() on a model like this yields:

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 28, 28, 6)         156       
_________________________________________________________________
average_pooling2d (AveragePo (None, 14, 14, 6)         0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 10, 10, 16)        2416      
_________________________________________________________________
average_pooling2d_1 (Average (None, 5, 5, 16)          0         
_________________________________________________________________
flatten (Flatten)            (None, 400)               0         
_________________________________________________________________
dense (Dense)                (None, 120)               48120     
_________________________________________________________________
dense_1 (Dense)              (None, 84)                10164     
_________________________________________________________________
dense_2 (Dense)              (None, 10)                850       
=================================================================
Total params: 61,706
Trainable params: 61,706
Non-trainable params: 0
_________________________________________________________________

The pooling layers have no trainable parameters. Maybe those parameters aren't so important for performance, but I'm curious how to implement the original pooling layers in Tensorflow.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.