"Bayesian inference treats model parameters as random variables whereas frequentist inference considers them to be estimates of fixed, ‘true’ quantities." (Ellison 2004)

What does it mean in the Bayesian methodology that parameters are random? And how does this differ from Maximum Likelihood estimation where they are fixed?

Isabella Ghement's response in this thread is a good start, but I still don't completely understand the "randomness" of the parameters.

  • 1
    $\begingroup$ The randomness is commonly explained by pointing out that the parameters are unknown and hence with some probability is equal to some specific (but unknown) value. This is then formalized by the prior distribution that specifies with what probability the parameters take on a certain value. But perhaps you could be more specific why or how the "randomness" puzzles you? How is that you do not find the other answer satifying? $\endgroup$ – Jesper for President Dec 12 '19 at 2:24

If you allow me, for a moment forget about the frequentist v/s Bayesian style of things and let us simply dive into what are we "modeling".

Imagine that all samples $x$ in a dataset belong to some set $\mathcal{X}$ and the probability of observing a sample $X = x$ is $p(X = x)$. (Note the usage of capital letter for a random variable and $x$ a realization of $X$). If you don't like talking in probabilities, think of this as just assigning some positive "score" to every sample. Hence, if something doesn't belong in $\mathcal{X}$, it's score is zero. As an example, if $\mathcal{X}$ were the set of all kinds of cats and dogs, $p (X = \text{any kind of tiger}) = 0$.

The goal of all machine learning is to estimate this object $p(X)$, why? Because if you could truly achieve this goal of getting the true $p(X)$, you could do all kinds of things - generate all possible cats and dogs (by sampling from $p(X)$), check if something is a cat or a dog and so on. Here I have subsumed things like classification labels $Y$ etc. into $X$ as they don't matter for this discussion.

If you are on board with this, the next step is to actually "learn" $p(X)$. We make modeling choices and assumptions to make this problem computationally tractable. I make the following choice to explain this distribution -

$p(X)$ can be explained by certain hidden parameters $\theta$ which are jointly modeled as $p(X, \theta) = p(\theta) p(X | \theta)$

Is my choice correct? Who knows? But I made this modeling choice. How do we recover $p(X)$ from this? By simply using the rule of probability to marginalize out all the possible parameters $\theta$, $p(X) = \int p(X, \theta) d\theta$. Whether this is possible in practice, is a different question altogether, but we have hopefully devised a model which can allow us to do that.

Our sole purpose now after making our modeling choice explicit is just to find that $\theta$ which will "explain" the sample(s) we have seen. One way to "explain" the sample(s) is to measure "likelihood" $p(X|\theta)$ - higher likelihood means better fit under the modeling assumptions (note again it doesn't have anything to do with reality, it is the best under the assumptions we chose to stick with). Hence, we must maximize $p(X|\theta)$ and find the best one, say $\theta^\star$.

$\theta^\star = \underset{\Theta}{\text{argmax} } p(X | \theta)$

This is Maximum Likelihood Estimation (MLE). This estimator claims that there is one best $\theta^\star$ (the result of your optimization process) which can be used to explain everything about your data.

Now comes the Bayesian and tells you, I don't want to put all eggs in one basket and make sure I don't miss out on other possible $\theta$'s. A Bayesian is uncertain and always quantifies uncertainties. And hence says, I will use the Bayes theorem to build a probability distribution around all possible $\theta$'s to quantify my uncertainty as

$p(\theta | X) = \frac{p(X | \theta) p(\theta)}{p(X)}$

In simple words, a Bayesian is assigning scores to all possible $\theta$'s without ever claiming what is the true one. You might ask how do we know what is the true one? We then further use decision theory to pick a $\theta^\star$ based on some risk function (a discussion for another time). And hence, a Bayesian treats $\theta$ as a random variable (and therefore a probability distribution to quantify it). $p(\theta)$ is the prior - what the Bayesian "believes" could be the true values before seeing any sample and $p(\theta | X)$ is what the Bayesian "believes" after seeing the sample.

How do you go from being a Bayesian to a frequentist? Simply find the "mode" of $p(\theta | X)$ - a $\theta^\star$ which gives you the best score to "explain" $X$. This problem then becomes

$\theta^\star = \underset{\Theta}{\text{argmax} } p(X | \theta) p(\theta)$

(notice we don't need $p(X)$ as it is independent of $\theta$). This is known as Maximum a-posteriori Estimation (MAP). If you have no strong feelings about the prior, you can imagine using a uniform prior where you initially believe that all $\theta$'s are equally likely to be $\theta^\star$. Assuming that, $p(\theta) \propto 1$ can also be dropped out of the optimization as it becomes a constant and you recover Maximum Likelihood Estimation (MLE).

In conclusion, both schools are modeling the same things, Bayesians are just explicit about their choices.

  • $\begingroup$ This was very helpful, thank you! $\endgroup$ – EagleEye Dec 13 '19 at 17:48

Here is my answer: Think about the data generation process. Suppose we have a random variable $X$ depending on a parameter $\theta$. We assume $\theta$ follows some distribution as in the Bayesian philosophy.

In this case, consider the joint distribution of $(X, \theta)$. Each time $X$ is generated, a $\theta$ is generated as well. However, you cannot see the value of $\theta$.

So in this way, both $X$ and $\theta$ are random. This is how I understand it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.