Where does advanced probability theory/stochastics come into machine learning? I have a chance of writing my bachelor thesis about ML in our Probility Theory/Stochastics deparment at the university.
In the search of interesting theoretical topics I am studying Bishops Pattern Recognition and Machine Learning book.
I have read into most of the chapters now and the only place where I obviously see some advanced probability theory (stochastic processes) is in the Gaussian processes chapter.
Are there any other areas which I overlooked or where the advanced math isn't so obvious s.t. I could find interesting theoretical questions regarding probability theory or stochastic analysis?
 A: For a very probabilistic application of ML, take a look at Markov Fields (undirected graphical models) that are used in image processing.
For a more stochastic-processes flavor, it would be interesting to study the path-properties of the test vs training MSE for a simple machine learning model applied to gaussian data (e.g., K-nearest neighbors under a gaussian mixture vs single multivariate gaussian). For example:


*

*The training error will be a supermartingale wrt model complexity (i.e., fewer neighbors = more complexity)

*However, the test error will generally be convex or quasi-convex (on average)


It would be interesting to examine the stochastic properties of these two paths (i.e., under repeated sampling from the underlying distribution). In particular, how well can one identify the minimum of curve 2? (Compare this to the "1-standard deviation rule", where you find the empirical minimum and then reduce the complexity until the test error curve intersects the line located one standard deviation above the minimum empirical test error).
Just some thoughts. 

Someone just asked a question about Markov Fields here: Modeling a Classification Problem with an Undirected Graphical Model
That should give you some more ideas.
A: There is an on-line variant of random forests whose training and justifications are based on a stochastic process (the Mondrian process).


*

*Mondrian Forests: Efficient Online Random Forests contains material about the random forest.

*The Mondrian Process contains more theoretical background about the underlying process.
