1
$\begingroup$

I use 2 layer CNN network for NLP task with triplet loss with margin 0.2. The task is to learn document embeddings to find similar docs. My architecture is similar to this https://arxiv.org/abs/1406.3830

I use truncated_normal init function from TensorFlow to initialize conv weights. I also use AdamOptimizer with default params. Then I subsample small (or big) dataset and use 5-10 epochs to train on it. But the loss stays close to 0.2 all the time. I am defiantly underfitting. But the underfitting is not related to insufficient number of layers because same architecture works fine in literature. The problem I guess is the initialization function of convolution layers weights - it is just stuck in some local minimum. But I don't know for sure.

Where is my mistake? What is a good way to init my conv layers?

$\endgroup$
  • $\begingroup$ Does your network converge and overfit if you make the training set just one or a few datapoints? That is often a good sanity check. $\endgroup$ – shimao Aug 24 '17 at 17:37
  • $\begingroup$ no, it did not. The problem was my mistake, when I fixed it the model started to overfit as expected. You are right $\endgroup$ – guyos Aug 24 '17 at 17:46
1
$\begingroup$

The problem was my mistake. I did not compose triples properly, there was no anchor, positive and negative examples, they were all anchors or positives or negatives. My mistake. That is why neural net could not find no way to optimize. After I fixed this mistake the loss went down as expected and it started to overfit, luckily.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.