1
$\begingroup$

not quite a math-question, but I have a doubt.

I'm trying to build from scratch the Pix2pix network, on the facades dataset, and I think I finally got a good model (from the paper I borrowed just the idea, the model structure I'm trying to find it myself), however, if all the previous trials had some artifacts that was imputed to the discriminator, not I think it's working fine, except for some artifacts:

enter image description here

Those are:

  • Ground truth
  • segmentation
  • Model generation

As you can see the right and bottom border have some artifacts, that I'm trying to find a fix for them

the generator is built like this (UNet):

  • series of 2D conv 3x3 kernel, 2x2 strides, padding same (with batch norm before activation)
  • series of 2D transpose conv 3x3 kernel, 2x2 strides, padding same (with residual connection from the encoder, just like UNet)
  • some 2D transpose conv with various filters, to let the model still additionally change the decoder reconstruction

the decoder instead is a fully convolutional net with kernel 3x3 and strides 2x2

Now, I'm using "padding same" everywhere, that was my initial though on why those happens, however I changed to "same" and they are still here... at the moment, I think that this might be due to the different shapes of strides and kernels, however this idea don't quite hold

Any idea on what can be the artifact origin?

Code: I leave the code for the 2 models, so that i maybe easier to understand.

Generator:

class TBNRConv(K.layers.Layer):
    def __init__(self, filters=128, kernel_size=(3, 3),strides=(1, 1),padding='same', use_bias=False, act=tf.nn.leaky_relu):
        super().__init__()
        self.conv = K.layers.Conv2DTranspose(filters=filters, kernel_size=kernel_size,strides=strides,padding=padding, use_bias=use_bias)
        self.bn = K.layers.BatchNormalization()
        self.act = act
    def call(self, inputs, *args, **kwargs):
        x = self.conv(inputs)
        x = self.bn(x)
        x = self.act(x)
        return x

class BNRConv(K.layers.Layer):
    def __init__(self, filters=128, kernel_size=(3,3), strides=(2,2), padding="same", use_bias=False, act=tf.nn.leaky_relu):
        super().__init__()
        self.conv = K.layers.Conv2D(filters=filters, kernel_size=kernel_size, strides=strides,padding=padding,use_bias=use_bias)
        self.bn = K.layers.BatchNormalization()
        self.act = act
    def call(self, inputs, *args, **kwargs):
        x = self.conv(inputs)
        x = self.bn(x)
        x = self.act(x)
        return x

image = K.layers.Input(shape=x_train.shape[1:])
e_1 = BNRConv(strides=(1,1))(image)
e_2 = BNRConv(filters=64)(e_1)
e_2 = BNRConv(strides=(1,1))(e_2)
e_3 = BNRConv(filters=64)(e_2)
e_3 = BNRConv(strides=(1,1))(e_3)
e_4 = BNRConv(filters=64)(e_3)
e_4 = BNRConv(strides=(1,1))(e_4)
bottleneck = BNRConv()(e_4)

d_4 = TBNRConv(strides=(2,2))(bottleneck)
d_4 = TBNRConv(strides=(1,1))(d_4)
d_4 = K.layers.Add()([d_4, e_4])

d_3 = TBNRConv(strides=(2,2))(d_4)
d_3 = TBNRConv(strides=(1,1))(d_3)
d_3 = K.layers.Add()([d_3, e_3])

d_2 = TBNRConv(strides=(2,2))(d_3)
d_2 = TBNRConv(strides=(1,1))(d_2)
d_2 = K.layers.Add()([d_2, e_2])

d_1 = TBNRConv(strides=(2,2))(d_2)
d_1 = TBNRConv(strides=(1,1))(d_1)
d_1 = K.layers.Add()([d_1, e_1])

x = TBNRConv(strides=(1,1), kernel_size=(1,1), filters=64)(d_1)
x = TBNRConv(strides=(1,1), kernel_size=(3,3), filters=64)(x)
x = TBNRConv(strides=(1,1), kernel_size=(5,5), filters=64)(x)
x = TBNRConv(strides=(1,1), kernel_size=(3,3), filters=64)(x)
x = TBNRConv(strides=(1,1), kernel_size=(1,1), filters=64)(x)
x = K.layers.Conv2DTranspose(filters=3, kernel_size=(3, 3), padding='same',use_bias=False, activation='tanh')(x)
generator = K.Model(inputs=[image], outputs=[x])

Discriminator:

image = K.layers.Input(shape=x_train.shape[1:])
reference = K.layers.Input(shape=x_train.shape[1:])

y = K.layers.Conv2D(filters=32, kernel_size=(3,3), activation=tf.nn.leaky_relu, padding="SAME")(reference)
y = K.layers.Conv2D(filters=32, kernel_size=(3,3), activation=tf.nn.leaky_relu, strides=(2,2), padding="SAME")(y)
y = K.layers.Dropout(0.3)(y)
y = K.layers.Conv2D(filters=32, kernel_size=(3,3), activation=tf.nn.leaky_relu, padding="SAME")(y)
y = K.layers.Conv2D(filters=16, kernel_size=(3,3), activation=tf.nn.leaky_relu, strides=(2,2), padding="SAME")(y)

x = K.layers.Conv2D(filters=32, kernel_size=(3,3), activation=tf.nn.leaky_relu, padding="SAME")(image)
x = K.layers.Conv2D(filters=32, kernel_size=(3,3), activation=tf.nn.leaky_relu, strides=(2,2), padding="SAME")(x)
x = K.layers.Dropout(0.3)(x)
x = K.layers.Conv2D(filters=32, kernel_size=(3,3), activation=tf.nn.leaky_relu, padding="SAME")(x)
x = K.layers.Conv2D(filters=16, kernel_size=(3,3), activation=tf.nn.leaky_relu, strides=(2,2), padding="SAME")(x)

z = K.layers.Concatenate()([x, y])
z = K.layers.Conv2D(filters=16, kernel_size=(3,3), activation=tf.nn.leaky_relu, strides=(2,2), padding="SAME")(z)
z = K.layers.Dropout(0.2)(z)
z = K.layers.Conv2D(filters=16, kernel_size=(3,3), activation=tf.nn.leaky_relu, strides=(2,2), padding="SAME")(z)
z = K.layers.Dropout(0.2)(z)
z = K.layers.Conv2D(filters=16, kernel_size=(3,3), activation=tf.nn.leaky_relu, strides=(2,2), padding="SAME")(z)
z = K.layers.Dense(1, activation="sigmoid")(z)
discriminator = K.Model(inputs=[image, reference], outputs=[z])
$\endgroup$

1 Answer 1

1
$\begingroup$

Torch implementations normally combat edge artifacts with 'reflect' padding, which is not present in TF. This question has some possible implementations in its answers section.

$\endgroup$
1
  • $\begingroup$ So, in other words, it “mirrors” the images? Along the boundaries of the channles $\endgroup$
    – Alberto
    Aug 4, 2022 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.