I'm trying to understand how cross-entropy works for finding the optimal weights in neural networks. According to Eli Bendersky's website and neural networks and deeplearning tutorial, we can find the optimal weights using gradient descent. What if we use the cross-entropy loss function without using gradient descent? I mean is there any way to just find the derivative of cross-entropy with respect to w_ij and then makes the result of derivative equal to zero like what we do in Winer. Is there any resource available which try to find the optimal weights of neural networks using this way?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
First, remember that gradient descent is just a numerical technique for optimizing differentiable functions, which often means finding a local minimum or a place where the gradient of the function is effectively zero.
In most cases, the equation $\nabla f =0$ cannot be solved symbolically. So even once you've derived an express for the gradient, you will still need a numerical technique to find a root. One exception is a linear neural network, which is ultimately linear regression, and therefore you could reverse engineer a set of optimal weights from the least-squares solution.
