1
$\begingroup$

I'm implementing an Inception module from scratch, specifically this version with dimensionality reduction, and I'm not sure about how to calculate the input error delta when doing the backpropagation pass.

This is the layer/module structure:

enter image description here

Now, what I'm doing right now is:

  • During the forward pass, keep track of the activity/activation for each intermediate step
  • Compute the input error delta for each of the 4 "pipelines" individually (using the intermediate activity values when needed from step 1)
  • Sum the 4 input error delta tensors (one for each pipeline) to get the global input error delta for the whole module

I'm not sure about the last step though, so my questions are:

Is it right to just sum the 4 input error deltas or should them be averaged together according to some criteria? Any other advice or something I'm doing wrong here?

I couldn't find anything about it on the internet, everyone is just using TensorFlow or some other library that has automatic differentiation, so I'm not sure about what's exactly happing there behind the scenes.

Thank you for your help!

$\endgroup$
0
$\begingroup$

Forgot to post an answer, yeah the gradients to the input layer should just be summed together, as the inputs were just propagated independently to each separate channel. Here is my implementation of this inception layer in C#/cuDNN, as a reference.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.