# I am trying to build a progressive auto encoder neural network and I am not sure how to discard old weights?

The goal of the network is simple, encode and decode images at a smaller scale and slowly increasing the network complexity, the input image size and its output quality.

My current weights for my Encoder network structure before growth is below.

3) Convolution [512, 1, 1, 1] # [output_channel, input_channel, kernel_sizes]
2) BatchNormalization [512]
1) ReLU [512]
0) Convolution [64, 512, 4, 4]


My weights after growing my Encoder network by 1 layer.

6) Convolution [256, 1, 1, 1] # [output_channel, input_channel, kernel_sizes]
5) BatchNormalization [256]
4) ReLU [256]
3) Convolution [512, 256, 4, 4]
2) BatchNormalization [512]
1) ReLU [512]
0) Convolution [64, 512, 4, 4]


As you can see from above no actions are required for layer 0, 1, 2's weight in the second state. However, my input is a grayscale image so I have to modify layer 3's input channel before adding another layer on top. After growth, I would go from 1 channel to 512 channels to 1 channel to 256 channels, then 256 to 512 channels. Now I will train the 1st state for x amount of epochs before moving on. Which means layer 3's weight from the first state will be trained. This leads to my question, what am I suppose to do with it? I have to discard it and replace it with new a convolution size of [512, 256, 4, 4] so my model can still work correctly. The previous weights are trained should I just add those weights back into the new replacement convolution? Should I duplicate 255 channels and keep it? Or just don't worry about?

Edited: My bad the BatchNormalization values are actually different, so weights 0 and 1 remains unchanged.