Where does advanced probability theory/stochastics come into machine learning?

I have a chance of writing my bachelor thesis about ML in our Probility Theory/Stochastics deparment at the university.

In the search of interesting theoretical topics I am studying Bishops Pattern Recognition and Machine Learning book.

I have read into most of the chapters now and the only place where I obviously see some advanced probability theory (stochastic processes) is in the Gaussian processes chapter.

Are there any other areas which I overlooked or where the advanced math isn't so obvious s.t. I could find interesting theoretical questions regarding probability theory or stochastic analysis?

• What do you consider "advanced" probability theory and stochastics? Can you outline your class syllabus?
– user75138
Oct 14 '15 at 13:18
• @Bey: I had "Intr. to Prob. Theory" where you just get a rigorous introduction to PT (CLT, LLG, characteristic functions, the stuff you need), then stochastic processes where we talked about cond. expectation, markov chains, brownian motions and next semester (still before I write my bachelor thesis) I'll have introduction to stochastic analysis where we will do martingales and Itô calculus. For me, advanced means anything that builds upon these themes or is related to them Oct 14 '15 at 13:27
• Looking at applied areas is generally not a good way of finding interesting theoretical topics. Its not machine learning but statistics - survival analysis, but apparently a lot of the proofs were very much improved by using counting processes (eg ms.uky.edu/~mai/sta709/topic.pdf) Oct 15 '15 at 17:38

For a very probabilistic application of ML, take a look at Markov Fields (undirected graphical models) that are used in image processing.

For a more stochastic-processes flavor, it would be interesting to study the path-properties of the test vs training MSE for a simple machine learning model applied to gaussian data (e.g., K-nearest neighbors under a gaussian mixture vs single multivariate gaussian). For example:

1. The training error will be a supermartingale wrt model complexity (i.e., fewer neighbors = more complexity)
2. However, the test error will generally be convex or quasi-convex (on average)

It would be interesting to examine the stochastic properties of these two paths (i.e., under repeated sampling from the underlying distribution). In particular, how well can one identify the minimum of curve 2? (Compare this to the "1-standard deviation rule", where you find the empirical minimum and then reduce the complexity until the test error curve intersects the line located one standard deviation above the minimum empirical test error).

Just some thoughts.

Someone just asked a question about Markov Fields here: Modeling a Classification Problem with an Undirected Graphical Model

That should give you some more ideas.

There is an on-line variant of random forests whose training and justifications are based on a stochastic process (the Mondrian process).