12
$\begingroup$

For the past year, I've been hearing a lot about Probabilistic Programming (PP) frameworks like PyMC3 and Stan, and how great PP is. And today, someone shared this link with me: Pyro: a Deep Probabilistic Programming Language

However, I don't really follow what is special about it since it feels like whatever you can do in PP you can do in any other general purpose language. I'm sure there are technical aspects about PP that make it attractive (e.g. parallel computing), but this aside, is PP really any different from any other language?

Question: I was wondering if there was a consensus of what PP was and how it differs from other statistical-focused software like R, Matlab, Mathematica. It should be noted that PyMC3 and Stan are focused on more Bayesian analysis.

Doing a little research on Google, I came across the following two definitions. The first more abstract, and the second more about PP's technical characteristics.

1.2. Probabilistic Programming Is

Instead, probabilistic programming is a tool for statistical modeling. The idea is to borrow lessons from the world of programming languages and apply them to the problems of designing and using statistical models. Experts construct statistical models already—by hand, in mathematical notation on paper—but it's an expert-only process that's hard to support with mechanical reasoning. The key insight in PP is that statistical modeling can, when you do it enough, start to feel a lot like programming. If we make the leap and actually use a real language for our modeling, many new tools become feasible. We can start to automate the tasks that used to justify writing a paper for each instance.

Here's a second definition: a probabilistic programming language is an ordinary programming language with rand and a great big pile of related tools that help you understand the program's statistical behavior.

Both of these definitions are accurate. They just emphasize different angles on the same core idea. Which one makes sense to you will depend on what you want to use PP for. But don't get distracted by the fact that PPL programs look a lot like ordinary software implementations, where the goal is to run the program and get some kind of output. The goal in PP is analysis, not execution (added emphasis).

-- Probabilistic Programming

I'd like to know if the general statistical community agrees with these two definitions of PP, and if there are any other characteristics this definition may be missing.

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ I would agree with the first definition: PP makes you define a statistical model and handles the simulation part on its own. Examples besides Stan are BUGS, Church, Anglican. R is not a PP. $\endgroup$
    – Xi'an
    Nov 6, 2017 at 18:12
  • $\begingroup$ @Xi'an, would you say that PP seems to focus mostly on Bayesian statistical modeling? If so, is PP meant to only support a Bayesian approach? $\endgroup$
    – Jon
    Nov 7, 2017 at 16:56
  • $\begingroup$ The focus is on "Hierarchical modeling". This is inherently convenient for Bayesian methods. Although there is a less natural frequentist interpretation as well. $\endgroup$
    – knrumsey
    Nov 10, 2017 at 20:42
  • $\begingroup$ @Jon: I suppose the answer is "primarily but not entirely". As a whole PP focuses on using probability theory to model uncertainty and make predictions. That doesn't necessitate the use of Bayes's theorem. For example, Monte Carlo methods themselves are frequentist procedures. Yesss, we use MC methods to sample Bayesian posteriors usually but that doesn't mean that MC methods only support Bayesian inference. $\endgroup$
    – usεr11852
    Feb 24, 2023 at 19:18

1 Answer 1

Reset to default
2
$\begingroup$

Probabilistic Programming is a technique for defining a statistical model. Unlike defining a model by its probability distribution function, or drawing a graph, you express the model in a programming language, typically as a forward sampler.

Automatic inference from a model specification is a typical feature of probabilistic programming tools, but it is not essential, and there is no need for it to be Bayesian. There are a variety of useful things that you can do with a model specified as a probabilistic program. For example, the paper Deriving Probability Density Functions from Probabilistic Functional Programs describes a tool that analyzes a probabilistic program and works out its probability distribution function. The paper Detecting Parameter Symmetries in Probabilistic Models analyzes a probabilistic program for parameter symmetries. This kind of work also falls under probabilistic programming.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.