Building a neural network with two training paths in Keras I am trying to build a NN in Keras with two different output paths where the first path informs the second. The first path passes its loss to the end of the second path, like so:


*

*Pass through layer A then layer B, calculate loss as $L(Y_1, \hat{Y_1})$ and back-propagate error. (easy)

*Pass through layer A then layer C, calculate loss incorporating the loss from step 1 as $L(Y_2, \hat{Y_2}) - \lambda L (Y_1, \hat{Y_1})$, and back-propagate error. (not sure how to do this)


I think I need to save the error from step 1, and use it to build a customized loss function for step 2. Any guidance would be appreciated.
 A: Try using a custom training loop. Assuming you are using Tensorflow 2:
for epoch in range(num_epochs):
    for x, y1, y2 in train_dataset:
        with tf.GradientTape(persistent=True) as tape:
            output1, output2 = model(x, training=True)
            loss1 = calc_loss_1(output1, y1)
            loss2 = calc_loss_2(loss1, y2, output2)

        gradient1 = tape.gradient(loss1, model.trainable_variables)
        optimizer.apply_gradients(zip(gradient1, model.trainable_variables))

        # Make sure not to update B again by specifying layers to calculate grads for
        gradient2 = tape.gradient(loss2, [model.get_layer("C").trainable_variables,
                                          model.get_layer("A").trainable_variables])
        optimizer.apply_gradients(zip(gradient2, 
                                     [model.get_layer("C").trainable_variables,
                                      model.get_layer("A").trainable_variables]))

You can read more about custom training loops in the official Tensorflow docs. Beware in the above code snippet and in your use-case A will get updated twice by the gradients from loss1, and C does not contribute to loss1, so loss1 will not have any effect on its parameters.
