I'm looking to do an optical character recognition (OCR) project. After doing some research I've come across an architecture that seems interesting: CNN+RNN+CTC. I'm familiar with convoluted neural networks (CNN), and recurrent neural networks (RNN), but what is Connectionist Temporal Classification (CTC)? I'd like an explanation in layman's terms.

  • 4
    $\begingroup$ I recommend you to read the origin paper: Graves, et al. (2006) Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks, Proceedings of the 23 rd International Conference on Machine Learning (pdf). $\endgroup$
    – neo li
    Commented Mar 12, 2018 at 7:34

1 Answer 1


You have a dataset containing:

  • images I1, I2, ...
  • ground truth texts T1, T2, ... for the images I1, I2, ...

So your dataset could look something like that:

enter image description here

A Neural Network (NN) outputs a score for each possible horizontal position (often called time-step t in the literature) of the image. This looks something like this for a image with width 2 (t0, t1) and 2 possible characters ("a", "b"):

| t0 | t1 --+-----+---- a | 0.1 | 0.6 b | 0.9 | 0.4

To train such a NN, you must specify for each image where a character of the ground truth text is positioned in the image. As an example, think of an image containing the text "Hello". You must now specify where the "H" starts and ends (e.g. "H" starts at the 10th pixel and goes until the 25th pixel). The same for "e", "l, ... That sounds boring and is a hard work for large datasets.

Even if you managed to annotate a complete dataset in this way, there is another problem. The NN outputs the scores for each character at each time-step, see the table I've shown above for a toy example. We could now take the most likely character per time-step, this is "b" and "a" in the toy example. Now think of a larger text, e.g. "Hello". If the writer has a writing style which uses much space in horizontal position, each character would occupy multiple time-steps. Taking the most probable character per time-step, this could give us a text like "HHHHHHHHeeeellllllllloooo". How should we transform this text into the correct output? Remove each duplicate character? This yields "Helo", which is not correct. So, we would need some clever postprocessing.

CTC solves both problems:

  • you can train the network from pairs (I, T) without having to specify at which position a character occurs using the CTC loss
  • you don't have to postprocess the output, as a CTC decoder transforms the NN output into the final text

How is this achieved?

  • introduce a special character (CTC-blank, denoted as "-" in this text) to indicate that no character is seen at a given time-step
  • modify the ground truth text T to T' by inserting CTC-blanks and by repeating characters in in all possible ways
  • we know the image, we know the text, but we don't know where the text is positioned. So, let's just try all possible positions of the text "Hi----", "-Hi---", "--Hi--", ...
  • we also don't know how much space each character occupies in the image. So let's also try all possible alignments by allowing characters to repeat like "HHi----", "HHHi---", "HHHHi--", ...
  • do you see a problem here? Of course, if we allow a character to repeat multiple times, how do we handle real duplicate characters like the "l" in "Hello"? Well, just always insert a blank in between in these situations, that is e.g. "Hel-lo" or "Heeellll-------llo"
  • calculate score for each possible T' (that is for each transformation and each combination of these), sum over all scores which yields the loss for the pair (I, T)
  • decoding is easy: pick character with highest score for each time step, e.g. "HHHHHH-eeeellll-lll--oo---", throw away duplicate characters "H-el-l-o", throw away blanks "Hello", and we are done.

To illustrate this, have a look at the following image. It is in the context of speech recognition, however, text recognition is just the same. Decoding yields the same text for both speakers, even though alignment and position of the character differs.

enter image description here

Further reading:

  • 1
    $\begingroup$ thanks for the feedback. I've added an illustration about the dataset and improved the description. Further, the term time-step simply means "horizontal position" or x-coordinate in the context of text recognition. The term most likely originates from speech recognition (a similar task), where time-step refers to the position in time in the audio signal. $\endgroup$
    – Harry
    Commented Sep 4, 2018 at 16:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.