The goal of the network is simple, encode and decode images at a smaller scale and slowly increasing the network complexity, the input image size and its output quality.
My current weights for my Encoder network structure before growth is below.
3) Convolution [512, 1, 1, 1] # [output_channel, input_channel, kernel_sizes] 2) BatchNormalization  1) ReLU  0) Convolution [64, 512, 4, 4]
My weights after growing my Encoder network by 1 layer.
6) Convolution [256, 1, 1, 1] # [output_channel, input_channel, kernel_sizes] 5) BatchNormalization  4) ReLU  3) Convolution [512, 256, 4, 4] 2) BatchNormalization  1) ReLU  0) Convolution [64, 512, 4, 4]
As you can see from above no actions are required for layer 0, 1, 2's weight in the second state. However, my input is a grayscale image so I have to modify layer 3's input channel before adding another layer on top. After growth, I would go from 1 channel to 512 channels to 1 channel to 256 channels, then 256 to 512 channels. Now I will train the 1st state for x amount of epochs before moving on. Which means layer 3's weight from the first state will be trained. This leads to my question, what am I suppose to do with it? I have to discard it and replace it with new a convolution size of [512, 256, 4, 4] so my model can still work correctly. The previous weights are trained should I just add those weights back into the new replacement convolution? Should I duplicate 255 channels and keep it? Or just don't worry about?
Edited: My bad the BatchNormalization values are actually different, so weights 0 and 1 remains unchanged.