Proper Scoring Rule in Optical Character Recognition

Question

Cross Validated likes to promote proper scoring rules in "classification" problems. That is, get accurate probability predictions. Then make the classifications, taking into account the cost of misclassifications. This works well for a situation like medical diagnosis, where, as Frank Harrell argues, the physician would be most interested in the probability. Even the task of spam email detection, which a computer handles automatically by sticking emails in discrete buckets, has a role for a probability calibration.

One place where I cannot see a role for a well-tuned probability, however, is optical character recognition. We stick a printed document into a scanner, and it turns the images of the letters into text of the letters. As far as I can tell, all we would care about is how accurate the resulting text is.

1. What would be the role of a proper scoring rule that seeks the true probabilities if my lone goal is to train an accurate classifier?

2. Does optical character recognition do the statistics and decision in one step?

3. Why would I even want to use a proper scoring rule as the loss function if I'm just going to assess the model based on its ability to assign the highest class probability to the correct character?

Answer 1

First off, I wouldn't say it's CrossValidated that "likes to promote proper scoring rules". It's more a few very vociferous users. Present company not excepted.

I would agree that the role of scoring rules is much smaller in optical character recognition (OCR) than in many other domains, like medical diagnostics. The reason, IMO, is that the signal to noise ratio is much higher in OCR. We teach five-year-olds to read, after all. Nobody makes a conscious effort to obfuscate our classifiers. We rather make sure to display the signal in a standardized way (the address almost always goes in the same position on the envelope, pages are usually in portrait orientation etc.), and incentives are aligned with making classifiers' life easier. Finally, there is a very small number of target classes: 26 letters, 10 numerals.

In contrast, spammers have an incentive to obfuscate classifiers. In medical diagnostics, the true disease lurks somewhere deep in a highly complex human-shaped black box. Anything beyond the most trivial use cases (the common cold, which we can usually diagnose ourselves and don't visit the doctor with) thus is interpreted by highly trained professionals (either the meat or the silicon version). Image recognition, apart from toy examples, has a limitless number of possible classes to classify an image into.

In a high signal-to-noise situation like OCR on Western scripts, most instances will be probabilistically classified as one class with very high probability, and this classification will usually be correct. It's simply not very interesting to train a classifier to better probabilistically distinguish a lowercase g from a 9, because it's usually easy to do so well enough already, based on context.

So I would say that the emphasis on proper scoring rules is more important in low signal to noise situations. And conversely, I sometimes have the impression that people who rely on accuracy have learned classification in high signal to noise situations (like OCR), and may have difficulties with their toolset when this ratio changes in a new situation.