gradient descent in neural network Given that almost all the activation functions in neural networks are increasing, by the gradient descent rule, all parameters should be updated in the same direction (negative direction). Then how come we may have very large parameters?
 A: Neural network models work by superimposing sigmoidal (or other) activation functions to implement the desired mapping.  Consider the simple one-dimensional interpolation problem below:

There are four data points and they can be interpolated exactly using three sigmoid functions, each shown in a different colour.  In some cases, we need the sigmoid to increase as a function of the input variable, $x$, shown in red and blue, but sometimes we need one that is a decreasing function of $x$, shown in green.  We can achieve this in two ways, either by having a negative weight from the hidden layer neuron to the output layer neuron, or by having a negative weight from the input layer to the hidden unit.  This shows that minimising the cost function by gradient descent will lead to some parameters becoming positive and some negative, because the desired superposition of sigmoids is not strictly increasing.
Similar ideas arise in multi-dimensional settings as well, but they are more complicated to draw and to think about.  Also in practice the sigmoids won't all overlap in their flat regions, and we will get a more distributed representation of knowledge (for this problem, it ought to possible to interpolate the four datapoints using two sigmoids), but the same basic idea applies.
We get very large parameters quite easily if we need the ouptput of the network to change rapidly over a small change in $x$ (e.g. to make the sigmoid steeper), so we will need larger weights to magnify the small change in $x$ to produce a rapid change in $y$.  In this case the weights of the green hidden unit are likely to be larger than those of the red and blue units.
