I have a problem where gradients after minimization using Adam Optimizer only changes from hidden layer to output layer. But not from input layer to hidden layer, it stays the same as the previous training step (so gradients are not all 0s).
My neural network has the following architecture:
- Input layer: 150000 units (all with value either 0 or 1)
- Hidden layer: 150 units with dropout and ReLu
- Output Layer: 150000 units (all with value either 0 or 1)
- Loss Function: Sigmoid Cross Entropy
The task of the neural network is to match a GROUP of words from one language in another (I know there are other models, but I want to try out neural networks).
When I train using the above neural network. Recall is alright (33%) but Precision is too low (0.1%). At this moment, I think the problem is because gradients are not changed at all between the first layer and the hidden layer. Otherwise, I was wondering whether there are any other ways to improve this model.
I am also using tensorflow. So it might also be a coding problem