Implementing backpropagation in Theano I have been playing around with Theano for a while and read a lot of code examples and it looks like every time Theano graph needs to find gradients for a list of parameters it uses T.grad() function. Reading up on the grad() implementation, it seems that it is not doing backpropagation, but rather relies on chain rule and individual ops knowing how to differentiate themselves. 
Am I correct? How are the results different using this two methods? Does it make sense to implement backpropagation in Theano?
 A: You're correct that automatic differentiation (including theano's grad function) just uses the chain rule. The interesting point is that this is also how backpropagation works; it's just the chain rule. The standard backprop equations for computing the weight update direction amount to computing the gradient of the loss function with respect to the parameters.
The traditional way of training a network with backprop looks like this:


*

*Forward pass. Compute the activations and loss function, given the inputs and parameters. These will be used for computing the gradient in the next step.

*Backward pass. Compute the gradient of the loss function with respect to the parameters.

*Update parameters by moving in the direction opposite the gradient, with some step size.
So, traditional backprop training is just gradient descent. When you compute the gradient with theano, the result will be identical to what you'd get with the forward and backward passes using the backprop equations. Under the hood, theano is using reverse-mode automatic differentiation which, in the case of neural nets, is equivalent to the backprop equations (perhaps modulo some optimizations of the computational graph that theano might make to increase efficiency).
To complete the backprop training implementation, you'd just have to add some code for updating the parameters, given the gradient. Of course, there are many fancier modifications of the traditional gradient descent update rule, which you could use to boost performance.
