1
$\begingroup$

I want to merge two CNNs that are trained over the different dataset. I have taken two sequential models and merged them. But when using customized fit_generator, validation loss is not converging. How will I pass generators of different datasets?

input1_1 = keras.layers.Input(shape=(129,129,3))
x1 = keras.layers.Conv2D(kernel_size = (3,3), filters = 32, 
activation='PReLU')(input1_1)

x3 = keras.layers.MaxPooling2D(2,2)(x1)
x4 = keras.layers.Conv2D(kernel_size = (5,5), filters = 64, 
activation='relu')(x3)
x5 = keras.layers.MaxPooling2D(2,2)(x4)
x6 = keras.layers.Conv2D(kernel_size = (7,7), filters = 128, 
activation='relu')(x5)
d1_1 = keras.layers.Dropout(0.5)(x6)
br1_1= keras.layers.MaxPooling2D(2,2)(d1_1)
br1_1 = keras.layers.Flatten()(br1_1)


input2_2 = keras.layers.Input(shape=(129,129,3))
x1 = keras.layers.Conv2D(kernel_size = (3,3), filters = 32, 
activation='PReLU')(input2_2)

x3 = keras.layers.MaxPooling2D(2,2)(x1)
x4 = keras.layers.Conv2D(kernel_size = (5,5), filters = 64, 
activation='relu')(x3)
x5 = keras.layers.MaxPooling2D(2,2)(x4)
x6 = keras.layers.Conv2D(kernel_size = (7,7), filters = 128, 
activation='relu')(x5)
d2_2 = keras.layers.Dropout(0.5)(x6)
br2_2= keras.layers.MaxPooling2D(2,2)(d2_2)
br2_2 = keras.layers.Flatten()(br2_2)

added1_1 = keras.layers.concatenate([br1_1, br2_2], axis=1)
d2_3 = keras.layers.Dropout(0.5)(added1_1)
# d2_4 = keras.layers.Dropout(0.4)(d2_3)
out1_1 = keras.layers.Dense(159,activation='softmax',kernel_regularizer=regularizers.l2(0.01),
            activity_regularizer=regularizers.l1(0.01))(d2_3)
# model=keras.layers.Conv2DTranspose(kernel_size= (4,4), filters=10, activation='relu')(out)
modal1_1 = keras.models.Model(inputs=[input1_1,input2_2], outputs=out1_1)
modal1_1.summary()

I have tried to zip the generators(train_face, train_iris), but it is not doing the task.

def custom_iterator(Xp, Xs):

  from keras.preprocessing.image import ImageDataGenerator

  ig1 = ImageDataGenerator(rescale=1./255)
  ig2 = ImageDataGenerator(rescale=1./255)
  temp1 = ig1.flow_from_directory(Xp,target_size = (129, 129),batch_size = 
          10,class_mode = "categorical")
  temp2 = ig2.flow_from_directory(Xs,target_size = (129, 129),batch_size = 
          10,class_mode = "categorical")


  for batch in zip(temp1,temp2):


    yield [batch[0][0], batch[1][0]], [batch[0][1]]

After performing Zip if i call fit generator my validation loss is not converging.

train_data_dir = "C:\\Users\\Desktop\\SDUMLA\\faceDataset\\mixTrain"
validation_data_dir = "C:\\Users\\Desktop\\SDUMLA\\faceDataset\\mixTest"
train_data_dir_nir = "C:\\Users\\Desktop\\SDUMLA\\IRIS(129_129)\\mixTrain"
validation_data_dir_nir = "C:\\Users\\Desktop\\SDUMLA\\IRIS(129_129)\\mixTest"
train_gen=custom_iterator(train_data_dir_nir, train_data_dir)
valid_gen=custom_iterator(validation_data_dir_nir, validation_data_dir)
nb_train_samples=2226
nb_validation_samples=1272
batch_size=10
Spe=nb_train_samples/ batch_size
valiStep=nb_validation_samples / batch_size


print ('compiling')
modal1_1.compile(optimizer=keras.optimizers.Adam(lr=0.0001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False),
          loss='categorical_crossentropy',
          metrics=['accuracy'])

hist=modal1_1.fit_generator(train_gen,
           steps_per_epoch = Spe,
           epochs = 200,verbose = 1,
           validation_data = valid_gen,
           validation_steps = valiStep)

training results what i got during training

how can i deal with this problem? i have added dropouts to reduce overfitting. i don't know what mistake i am doing. can any one suggest the concept how i can merger two generators that are having different datasets, so it can serve my purpose.

$\endgroup$

closed as off-topic by Michael Chernick, kjetil b halvorsen, Peter Flom Jan 11 at 10:44

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – Michael Chernick, kjetil b halvorsen, Peter Flom
If this question can be reworded to fit the rules in the help center, please edit the question.