I'm currently working on a neural network for Handwritten Text Recognition (HTR) which expects images of words as input and outputs labels for those words. My HTR system is inspired by CRNN , which works quite well and contains:
- CNN for feature extraction
- RNN (LSTM) for sequential modelling and per-frame character predictions
- CTC as a loss function and to decode the per-frame predictions into the final label
Recently I came across a paper about sequence labelling with dilated convolutions . They used it in the domain of Natural Language Processing (NLP) to replace their bidirectional LSTMs. Only a few dilated convolution layers are needed to propagate information through all time-steps of the input sequence.
This paper kept me thinking: shouldn’t that also work in the domain of HTR? HTR and NLP have some similarities. Replacing the RNN part by such dilated convolutions, i.e. combining feature extraction and information propagation into the CNN part of the net. As I'm new to deep learning, I'm really looking forward to hear some feedback if it makes sense to invest time trying this out or if there is some reason why this will never work well. (Of course I'm also implementing a prototype, but this question should be seen from a theoretical point of view)