4
$\begingroup$

I am interested in machine learning; I have a particular liking for RNNs. I have coursework in some areas of computer science, e.g., data mining, optimization for ML algorithms, deep learning, and an undergraduate degree in mathematics. I have been applying ML algorithms for a while. I have some coursework in basic probability theory; enough to understand the sections in Goodfellow's Deep Learning book. I understand how most cost functions are related to probability density functions.

I was wondering; I am interested in deep learning with medical applications. Should I take a graduate level statistics course in Probability Theory that follows Durrett's textbook, or should I, for example, dive into Deep Learning papers and textbooks.

I was wondering if anyone could tell me what to do or, because this might be closed with such an opinion-based request, provide examples of how probability theory was (a) absolutely necessary in their own study (either making new algorithms or reading how others did) of machine or Deep Learning or (b) seemed to kind of be extra (for example, I feel this way to some extent about real analysis now; I took it a long time ago and although it increased my mathematical maturity, it is calculus I use constantly to backpropagate errors, not analysis, for example; it is possible to conceptualize and even develop learning algorithms only having intuitions about calculus and not having riogrous proofs such as those found in analysis). Don't get me wrong, I enjoyed analysis and would love to take all the courses I can, but my funding is eventually going to run out and I am expected to go into depth into a subject.

For example, I recently attended a conference with a very interesting presentation on RNNs that were using as input samples from Gaussian Processes used to model the missing data. It was designed by a statistician who presumably had Probability Theory. I would also like to design such pipelines. Btw, I am certainly taking graduate level Statistical Inference (this has been enthusiastically recommended twice by friends in statistics, while graduate level probability theory has been either dis-recommended or recommended without enthusiasm).

I hope this is specific enough, else happy to migrate to academia stack exchange.

$\endgroup$
7
$\begingroup$

As a statistics PhD student studying Bayesian deep learning and Gaussian processes, I have found it useful to be familiar with probability. I do not directly use the results for now because I am working on applied problems, but a lot of the theoretical work I look at is based on nonparametric techniques such as Gaussian or Dirichlet processes and those techniques are shown to have some properties using probability and functional analysis. Look for textbooks by Aad van der Vaart if you'd like an (extreme) example.

In other words, knowing probability theory opens up a lot of statistical literature for you to peruse.

If you care more about classification rates and less about uncertainty quantification, it might not be worth your time to get into the statistics. But then, if you are doing a logistic regression and the prediction of cancer for subject X is "1" (vs "0"), you might want to know how much confidence to place in that classification.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I am now thinking of studying a similar area - I was wondering how it has been for you and whether you have gone more towards probability as you have continued. I would not say I am great at probability, but I do enjoy pen and paper math. I also am planning to study Bayesian, so I was hoping to see what particular areas might merge pen and paper work with computing. $\endgroup$ – user0 Mar 29 at 15:45
  • 1
    $\begingroup$ I have branched out as my work has become more theoretical. In addition to probability concepts like convergence rates and martingales, tools from complex and functional analysis and Green's functions have become relevant. The fundamental mathematical fields are difficult but a good investment of your time. The material will always be valuable to you as a scientist, in contrast to the latest ML algorithm that people will forget about in a year. $\endgroup$ – daszil Mar 30 at 20:23
  • $\begingroup$ Appreciate the response. Since I posted this question I did indeed take probability theory, and I agree with you. I will look into those analysis topics. I am currently taking large sample and took stochastic processes last semester. I also really agree about the mathematical fields being useful for a scientist. $\endgroup$ – user0 Mar 30 at 21:02
0
$\begingroup$

Should I take a graduate level statistics course in Probability Theory that follows Durrett's textbook

NO. I have seen exactly zero instances where the sort of measure-theoretic background that is explored in depth in books like Durrett and Klenke is actually used in ML. The theory of probability and random variables can get very far, and cover all the concepts you actually see in an ML paper, without that. DeGroot & Schervish or Pishro-Nik (read all the way through) are enough. The measure-theoretic part comes in when you start needing to construct stochastic integrals or things like that but not likely in 99% of ML.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.