How does ResNet or CNN with skip connections solve the gradient exploding problem? I read some papers which said that the ResNet or Highway networks can mitigate the gradient vanishing/exploding problem in very deep neural networks. I'm not sure how the skip connections can solve the gradient exploding problem. Could anybody give some explanations or references? Thanks. 
 A: To my understanding, during backprop, skip connection's path will pass gradient update as well. Conceptually this update acts similar to synthetic gradient's purpose.
Instead of waiting for gradient to propagate back one layer at a time, skip connection's path allow gradient to reach those beginning nodes with greater magnitude by skipping some layers in between.
I personally do not find any improvement nor greater risk of encountering exploding gradient with skip connection.
A: I'm not 100% sure, but I would guess that this is more referring to normalization like BatchNorm rather than skip connections. It's not like ResNets will not explode without any normalization and not like plain VGG-style network will explode if you properly place BatchNorms. Skip connections, I guess, only help make the function smoother and the logic of the function that neural networks compute less convoluted, but it's pretty unrelated to exploding gradients problem.
I found that having an activation function after, for example, BatchNorm, may also be crucial to prevent exploding gradients. Sometimes, when I didn't have it follow BatchNorm, or when I had it precede BatchNorm loss was blowing up.
A: A possible reasoning is that residual connections reduce the feature space that a network searches for, as mentioned here:

A neural network without residual parts explores more of the feature space. This makes it more vulnerable to perturbations that cause it to leave the manifold, and necessitates extra training data to recover.

Due to lower sensitivity to perturbations, the network can tend to have smaller loss values, leading to smaller gradients and hence prevent gradient explosion.
