Which part will save time if training neural network using GPU? There are many work for training a deep neural network using GPU. From optimization perspective, which part saves time? Objective function evaluation? Gradient calculation? or something else? And and why?
Is Certain matrix operations (say matrix multiplication) faster in GPU than CPU?
In convex optimization Appendex C. We have flops count for each operation. In GPU are these changed? or GPU has much fast flops operations.

 A: You are right,GPUs, compared to CPUs (Central Processing Unit), are more specialized at performing matrix operations and other advanced mathematical transformations. GPUs obtain such a speed in operations because of the parallel/distributed architecture with many low-end processing nodes.
A: It's because neural networks are easy to parallel, for common network layers, the same operations are applied to the input and parameters, and the order of executing these operations doesn't affect the output.
GPUs are good at parallel processing as they have thousands of cores. AFAIK the GPU doesn't reduce the number of flops required for matrix operations compared with the CPU.
If a layer involves some operation that doesn't gain much from parallel processing (say the insertion sort), then using GPUs won't be so fast.
A: Neural networks are in general based on linear algebra computations. If you think about graphics operations like combining two textures (bitmaps), or changing somehow each pixel of texture, what GPUs are built for, it's nothing more and nothing less than simple matrix / vector operations.
Going deeper - GPU is processor that has thousands of computation units with different data registers and common command register, so for example, operation like:
X = X + 1

Where X is large matrix can utilize every single core of GPU as long as there is only single command to be run of each of them.
How particularly those features are use depends on software, usually linear algebra library / tool like for example Matlab or nVidia DNN libraries.
