Application of Bayes' theorem

Question

Background:

My professor gave us several copies of his hand-writing handouts last week. Due to the low quality of printer, there are several words in the handouts cannot be recognized. However, I found that I can "guess" what the professor has written on the handouts by reading context carefully.

Question:

1. Why could we guess and recognize vague words on the handouts?
2. Is it related to statistics?
3. If we find out how statistics works in the recognition process, can we apply it to other fields?

My idea:

Assumption:

• Define $0$ represents the recognizable degree we cannot recognize the words at all.
• Define $100$ represents the recognizable degree we can recognize the words easily.

A simple example:

                         I have ki_ _ed her.

In this sentence, "I","have","ki ed"and"her" is easy to recognize and they're 100. "_ _" represent two letters that we can not recognize and they're 0.

1. If we read the sentence in a romantic novel, it is reasonable to guess the two missing letters are "ss",that is, " I have kissed her."
2. If we read the sentence in a detective novel, it is reasonable to guess the two missing letters are "ll",that is, "I have killed her."

Bayes' theorem:

$P(ll \mid D)$= the probability that we find "kill" in the context of detective novel.

$P(ll) can be simply considered as the proportion of "kill" in a detective or a romantic novel.

$$P(ll \mid D)=\frac{P(D \mid ll)P(ll)}{P(D)}$$ $$P(ss \mid D)=\frac{P(D \mid ll)P(ll)}{P(D)}$$

Here $P(D)$ is a constant.

$$P(ll \mid D) \propto P(D \mid ll)P(ll)$$

That's it ! It's simple and natural.

What's the relationship between the background and the example The recognizable degree of vague words range from 0 to 100. The example,two of the letters are 0 and others are 100, is a extreme case.

My question:

1. Is there any mistakes or problem?
2. Could anyone provide any useful resources like articles, books etc to help me to go further on this topic?
3. I think is important to prove that human beings can naturally and automatically find $P(ll \mid D) \propto P(D \mid ll)P(ll)$ in their brains. How can I use statistics tools to prove that? Design an experiment?

---

Extension: Why could we read the words on the image below? How does our brain automatically correct the order of letters? I guess Bayes helps again.

Extension2: For many Chinese students, they have never learn traditional Chinese. However they can read a newspaper written by traditional Chinese easily. When they are asked to recognize a single tranditional Chinese character, they failure in most cases. Why?

Thank you!

Answer 1

This is a classical example in the field of Probabilistic Graphical Models (PGMs). PGMS are widely used in several areas, from text mining to bioinformatics, for problems such as inference and naturally belong within Machine Learning/pattern recognition/artificial intelligence.

As you have correctly suggested, the Bayes rule play a major role here. In essence, you can think of PGMs as a simplified representation of a very large joint distribution over many variables (simplified due to independence of variables), and some of the methods consist of repeatedly applying the Bayes rule.

A simple example of a PGM is a Hidden Markov Model (HMM), which is especially relevant to your example. Lucky for you, there is very good material for studying PGMs and even more on HMMs:

Prof. Daphne Koller from Stanford gives a free PGM course online, and I also recommend her book if you want a very rigorous (CS/math) treatment of the subject. For HMMs you can just google to find dozens of tutorials and select the one that fits your background the most.

Finally, PGMs are just one way of doing this kind of inference and if you want to expand your knowledge of such predictive methods you should look into the more general field of Machine Learning which I think you will find rich, deep and applicable (but maybe that's just me).

Answer 2

I used to think this idea "human beings can naturally and automatically find P(.|D) in their brains" was silly until I found this paper - figure 5 in the 2010 paper by Stigler given here Periods in history of statistics

Very simple physical machines can do fully valid Bayesian analyses - so maybe our brains can too.

Bitwise'a answer may be more direct for your feild of interest but almost anyone should be able to understand figure 5 ( maybe not people under 12 years of age ).

Below is an adaption of that figure I used in a talk, but it would be better to read the paper.

This quote from Pearl might also be helpful

Because this was the only biologically feasible way we could explain how the human brain deals with uncertainty, say when we read English text or cross a street. My colleagues at that time could not have cared less how humans do things, as long as the algorithm was efficient and correct. But as things turned out, it was this biologically inspired algorithm, called “belief propagation,” that scaled up (as an approximation) in practical applications ...

from http://magazine.amstat.org/blog/2012/11/01/pearl/