What is the target value for 2nd to last layer in neural network In a neural network implementing gradient descent, the following happens to the last layer and the second to the last layer. The cost function (average over all training examples of the sum of the squares of the differences between the predicted value and the target value) is calculated. Then the delta of the cost function is divided by the delta of any particular weight and this gives us the gradient. 
This is where I'm stuck. I have the following two questions: 


*

*Minimizing error and traversing down the curve in gradient descent is what we're after and this is obvious once we have this gradient vector. But what exactly do we do with the vector? For instance, if a particular weight has a value 0.5 and the gradient vector has a value of -0.1 for that weight, what do we do to that weight, just make it -0.4?

*When computing the gradient to begin with, it seems intuitive for the last layer and the second to layer because the target is known. But what exactly is the "target" for the 2nd to last layer? 

 A: Let's forget about neural nets for the moment and just think about gradient descent. This is a general-purpose optimization method that's not specific to neural nets. Say we have a cost function $f$ that assigns a cost to parameter vector $\theta$. The gradient evaluated at a given set of parameters is $\nabla f (\theta)$. This is a vector that points in the direction where $f$ is increasing most steeply. Gradient descent tries to decrease the cost by repeatedly stepping in the direction opposite the gradient, which is the direction of steepest decrease. The step size is controlled by the learning rate $\alpha$. On iteration $t$, the next set of parameters ($\theta_{t+1}$) is given by:
$$\theta_{t+1} = \theta_t - \alpha \nabla f(\theta_t)$$
For neural nets, the parameters are the weights, biases, etc. The way network layers show up is in the calculation of the gradient, which proceeds by the chain rule (yielding the familiar backprop expressions). All parameter updates are as shown above--there's no difference between layers.
Targets show up in the cost function for supervised learning problems. But, you shouldn't think about gradient descent in terms of targets. There are typically no targets for intermediate layers. And, in some (unsupervised) problems, there are no targets at all. Rather, gradient descent is a general purpose procedure that always proceeds as above, by stepping in the direction opposite the gradient of the cost function.
