Let's say I implemented a CNN. Is there an easy way I can validate, that my implementation of back-propagation does not contain errors ?
May be I can feed some dummy values into my network so it can do forward and backward pass, so I at the end I can compare the result with some pre-calculated target values ?
Or may be there is a way to validate the logic of my layers on layer-by-layer basis ? Is there a general solution for that ?
You can use dummy values, but calculating the targets may be cumbersome. And, you'd test with just the cases you thought of. A more reliable way to debug your back-propagation calculations is to use numerical gradients and do it several iterations. I mean, for each parameter $\theta$ in the network, you'll approximate $\frac{dL}{d\theta}$ as $\frac{L(\theta+\Delta\theta)-L(\theta)}{\Delta\theta}$ and compare with your analytical value ($L$ is your loss function). So, you're going to feed the network with the same input, but change parameter of interest slightly, get the output, and calculate the numerical derivative. There are other derivative approximation techniques by the way, but this is one of the simplest and serves your purpose.
Of course, there are some practical aspects:
you're going to choose a small $\Delta\theta$ (for ex. $0.1 \%$ of its current value) to get good approximation.
while comparing the analytical and numerical values, you won't get the same numbers. They'll be close but not perfectly equal. You can set a hard threshold, like $a=b$ if $|a-b|<10^{-6}$, or adaptive threshold depending on values of $a$ and $b$. Typically, if your calculation is wrong, you'll get very different results.
