Neural Network, gradients change only in one layer

Question

I have a problem where gradients after minimization using Adam Optimizer only changes from hidden layer to output layer. But not from input layer to hidden layer, it stays the same as the previous training step (so gradients are not all 0s).

My neural network has the following architecture:

• Input layer: 150000 units (all with value either 0 or 1)
• Hidden layer: 150 units with dropout and ReLu
• Output Layer: 150000 units (all with value either 0 or 1)
• Loss Function: Sigmoid Cross Entropy

The task of the neural network is to match a GROUP of words from one language in another (I know there are other models, but I want to try out neural networks).

When I train using the above neural network. Recall is alright (33%) but Precision is too low (0.1%). At this moment, I think the problem is because gradients are not changed at all between the first layer and the hidden layer. Otherwise, I was wondering whether there are any other ways to improve this model.

I am also using tensorflow. So it might also be a coding problem

Answer 1

A coding or a neural net design problem.. If you have a 3 layer network (input - hidden - output ) it will work for any kind of linear function. However the problem you try to tackle seams not a linear problem. (your network has many linear outputs since you dont have a single or dual output node, but in its essence its still a linear solver).

For example it wont work well for dividing groups on a mathematical 2D surface. So it wont work on advanced language stuff either, language is far more complex then any linear problem f(x)=y

Usually this is where deep nets kicks in, who have multiple hidden layers. And can recombine conclusions in each layer to solve more complex relations.

Answer 2

This seems to be a bit of parameter problem. The input space is 150.000 which is then mapped onto only 150, i.e. 0.1%, dimension. I don't think that these can capture the essence of the data and represent it well. Certainly not after only a few iterations. Also it's probably not a linear classification problem so you'll need more then one hidden layer.

So i would try to add some hidden units and a second or third hidden layer as well.

But for us to help you we'll need more information. Like e.g. the learning rate and the parameters of the Adam Optimizer.