Why does padding affect my results?

Question

I'm learning deep learning, and have been looking at a tutorial for autoencoders using MNIST. It's quite straightforward, and I think I understand it quite well so far. This is the code from the tutorial:

import numpy as np
import matplotlib.pyplot as plt
from keras.datasets import mnist
from keras.layers import Input, Dense, Conv2D, MaxPooling2D, UpSampling2D
from keras.models import Model
from keras import backend as K
#Load Dataset
(x_train, _), (x_test, _) = mnist.load_data()
#Scale Dataset values to lie between 0 and 1
x_train = x_train.astype('float32') / 255.

x_test = x_test.astype('float32') / 255.
x_train = np.reshape(x_train, (len(x_train), 28, 28, 1))
x_test = np.reshape(x_test, (len(x_test), 28, 28, 1))
#Add Noise to our MNNIST Dataset by sampling random values from Gaussian distribution by using np.random.normal() and adding it to our original images to change pixel values
noise_factor = 0.5
x_train_noisy = x_train + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_train.shape)
x_test_noisy = x_test + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_test.shape)
x_train_noisy = np.clip(x_train_noisy, 0., 1.)
x_test_noisy = np.clip(x_test_noisy, 0., 1.)

# Model Construction
input_img = Input(shape=(28, 28, 1))

x = Conv2D(32, (3, 3), activation='relu', padding='same')(input_img)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(32, (3, 3), activation='relu', padding='same')(x)
encoded = MaxPooling2D((2, 2), padding='same')(x)

# At this point the representation is (7, 7, 32)

x = Conv2D(32, (3, 3), activation='relu', padding='same')(encoded)
x = UpSampling2D((2, 2))(x)
x = Conv2D(32, (3, 3), activation='relu', padding='same')(x)
x = UpSampling2D((2, 2))(x)
decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)
autoencoder = Model(input_img, decoded)
autoencoder.compile(optimizer='adam', loss='binary_crossentropy')

autoencoder.summary()

This builds the following network:

Model: "model_3"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_3 (InputLayer)         (None, 28, 28, 1)         0         
_________________________________________________________________
conv2d_11 (Conv2D)           (None, 28, 28, 32)        320       
_________________________________________________________________
max_pooling2d_5 (MaxPooling2 (None, 14, 14, 32)        0         
_________________________________________________________________
conv2d_12 (Conv2D)           (None, 14, 14, 32)        9248      
_________________________________________________________________
max_pooling2d_6 (MaxPooling2 (None, 7, 7, 32)          0         
_________________________________________________________________
conv2d_13 (Conv2D)           (None, 7, 7, 32)          9248      
_________________________________________________________________
up_sampling2d_5 (UpSampling2 (None, 14, 14, 32)        0         
_________________________________________________________________
conv2d_14 (Conv2D)           (None, 14, 14, 32)        9248      
_________________________________________________________________
up_sampling2d_6 (UpSampling2 (None, 28, 28, 32)        0         
_________________________________________________________________
conv2d_15 (Conv2D)           (None, 28, 28, 1)         289       
=================================================================
Total params: 28,353
Trainable params: 28,353
Non-trainable params: 0

So far, so good. I can train it with

autoencoder.fit(x_train_noisy, x_train, 
    epochs=10,
    batch_size=128,
    shuffle=True
)

and it converges to a solution with a loss of approx. 0.099 (and roughly the same on the test set).

But now I tried to pad my input images. I did that in order to simplify building deeper autoencoders, but that's not the point here, the point is that even by just padding the images slightly, and keeping everything else constant, the results changed quite a bit:

x_train_padded = np.array([np.pad(x, ((2, 2), (2, 2), (0, 0)), mode='edge') for x in x_train])
x_test_padded = np.array([np.pad(x, ((2, 2), (2, 2), (0, 0)), mode='edge') for x in x_test])

x_train_padded_noisy = x_train_padded + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_train_padded.shape)
x_test_padded_noisy = x_test_padded + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_test_padded.shape)
x_train_padded_noisy = np.clip(x_train_padded_noisy, 0., 1.)
x_test_padded_noisy = np.clip(x_test_padded_noisy, 0., 1.)


# Model Construction
input_img_padded = Input(shape=(32, 32, 1))

x = Conv2D(32, (3, 3), activation='relu', padding='same')(input_img_padded)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(32, (3, 3), activation='relu', padding='same')(x)
encoded = MaxPooling2D((2, 2), padding='same')(x)

# At this point the representation is (8, 8, 32)

x = Conv2D(32, (3, 3), activation='relu', padding='same')(encoded)
x = UpSampling2D((2, 2))(x)
x = Conv2D(32, (3, 3), activation='relu', padding='same')(x)
x = UpSampling2D((2, 2))(x)
decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)
autoencoder_padded = Model(input_img_padded, decoded)
autoencoder_padded.compile(optimizer='adam', loss='binary_crossentropy')

autoencoder_padded.summary()

As you can see, the network is essentially the same as before (same convolution layers, same total number of parameters):

Model: "model_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_4 (InputLayer)         (None, 32, 32, 1)         0         
_________________________________________________________________
conv2d_16 (Conv2D)           (None, 32, 32, 32)        320       
_________________________________________________________________
max_pooling2d_7 (MaxPooling2 (None, 16, 16, 32)        0         
_________________________________________________________________
conv2d_17 (Conv2D)           (None, 16, 16, 32)        9248      
_________________________________________________________________
max_pooling2d_8 (MaxPooling2 (None, 8, 8, 32)          0         
_________________________________________________________________
conv2d_18 (Conv2D)           (None, 8, 8, 32)          9248      
_________________________________________________________________
up_sampling2d_7 (UpSampling2 (None, 16, 16, 32)        0         
_________________________________________________________________
conv2d_19 (Conv2D)           (None, 16, 16, 32)        9248      
_________________________________________________________________
up_sampling2d_8 (UpSampling2 (None, 32, 32, 32)        0         
_________________________________________________________________
conv2d_20 (Conv2D)           (None, 32, 32, 1)         289       
=================================================================
Total params: 28,353
Trainable params: 28,353
Non-trainable params: 0
_________________________________________________________________

However, when I train this network, using

autoencoder_padded.fit(x_train_padded_noisy, x_train_padded, 
    epochs=10,
    batch_size=128,
    shuffle=True
)

it achieves a loss of about 0.075, almost 25% less than before. Why is this? The only change was a couple of additional pixles around the edge of the image. Of course I'm happy for the improvement, but it really confuses me, and I'd like to understand why this happens.

By the way, I repeated the training several times, just in case there was some random effect that gave rise to the difference. But I got approx. the same results every time.

Answer 1

Actually, I think I figured it out. When the images are padded, the black area (the background) becomes larger. This area should be relatively easy for the autoencoder to learn how to recreate. Therefore, the average loss per pixel decreases, since the number of pixels increases but the difficulty does not increase.

This is also confirmed by noting that the total loss (loss x number of pixels) on the original images was 0.099*28* 28 = 77.6, while the total loss on the padded images was 0.075*32*32 = 76.8, so even though one loss value was larger than the other, the actual loss was more or less the same.

In other words, padding did not improve the autoencoder, but it modified the loss function, making the result at first seem better with padding than without.