Let's say I implemented a CNN. Is there an easy way I can validate, that my implementation of back-propagation does not contain errors ?

May be I can feed some dummy values into my network so it can do forward and backward pass, so I at the end I can compare the result with some pre-calculated target values ?

Or may be there is a way to validate the logic of my layers on layer-by-layer basis ? Is there a general solution for that ?


You can use dummy values, but calculating the targets may be cumbersome. And, you'd test with just the cases you thought of. A more reliable way to debug your back-propagation calculations is to use numerical gradients and do it several iterations. I mean, for each parameter $\theta$ in the network, you'll approximate $\frac{dL}{d\theta}$ as $\frac{L(\theta+\Delta\theta)-L(\theta)}{\Delta\theta}$ and compare with your analytical value ($L$ is your loss function). So, you're going to feed the network with the same input, but change parameter of interest slightly, get the output, and calculate the numerical derivative. There are other derivative approximation techniques by the way, but this is one of the simplest and serves your purpose.

Of course, there are some practical aspects:

  • you're going to choose a small $\Delta\theta$ (for ex. $0.1 \%$ of its current value) to get good approximation.

  • while comparing the analytical and numerical values, you won't get the same numbers. They'll be close but not perfectly equal. You can set a hard threshold, like $a=b$ if $|a-b|<10^{-6}$, or adaptive threshold depending on values of $a$ and $b$. Typically, if your calculation is wrong, you'll get very different results.

  • $\begingroup$ sorry, what do you mean by parameter $\theta$ ? A learnable parameter, like weight or bias ? $\endgroup$ – koryakinp Oct 18 '18 at 21:02
  • $\begingroup$ Yes, $w$ or $b$ are conceptually $\theta$s. $\endgroup$ – gunes Oct 19 '18 at 6:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.