In Unified Architecture for NLP paper time-delayed neural network proposed as a way to deal with variable length input. Input window slides over sequence and label each output with "time", then next layer takes max over time. But no explanation given how to back-propagate error through this max layer. We able to compute error only for specific "time", potentialy different for each max layer output node.

Reading original paper for TDNN didn't clarify anything. For me it looks like completly different NN architecture without any mention of max layer.

Can someone explain or point to explanation? My own googling skills failed.


Take a look at the paper Natural Language Processing (Almost) from Scratch. In the appendix it derives the gradient for the max layer. The short answer is that the input that produced the maximum value over the window receives the error gradient in its entirety, and the other inputs get an error of 0.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.