2
$\begingroup$

I was trying to understand model likelihood and now with great confidence I can also call it:

  • integrative likelihood
  • marginal likelihood
  • predictive likelihood
  • the evidence

since I read the paper:

enter image description here

The following formula:

$$ \begin{align} p(X\mid M)=\int_\theta p(X \mid \theta, M) \,p(\theta \mid M)\, d\theta \tag{1} \end{align} $$

is marginalized out parameters formula.

This formula say when $\theta$ vary, we will get model likelihood given $X$, since $X$ is fixed (all data we have).

But, the shorter formula also called evidence would be:

$$ \begin{align} p(X)=\int_\theta p(X \mid \theta) \,p(\theta)\, d\theta \tag{2} \end{align} $$

This leads me to a conclusion that if not explicitly set we always assume $p(X)$ is $p(X \mid M)$ because we need to have the model always.


Question:

We never write $p(M \mid \theta)$ so we never write the model depends on parameters because the model is not a random variable I guess. It is just a parameter from the finite/infinite set of models.

Or maybe the model is a random variable after all. Can you elaborate and help me sort how parameters are connected with the model.

As a preparation to this question I have several similar treads in here and I am actively reading 2 books on Bayesian subject but this is still unknown to me.

$\endgroup$
13
  • $\begingroup$ Can you precise what do you mean by the Model ? Is it the model that you are selecting and that contains the parameters theta (linear reg, trees, neural net etc...) or is it the underlaying Model that generated X ? $\endgroup$
    – Romain
    Mar 27, 2021 at 10:48
  • $\begingroup$ They’re referring to the former. (This seems to be a follow-up to their question here.) $\endgroup$ Mar 27, 2021 at 12:14
  • $\begingroup$ @GoodLuck, let me ask you this: what do you think is the set of options for the random variable $M$? If there’s only one called $m$, then to be a well-defined distribution, $p(m \mid \theta)$ must be 1 for every value $\theta$ takes on. $\endgroup$ Mar 27, 2021 at 12:40
  • 1
    $\begingroup$ Look at keywords like "Bayesian model selection", "Bayes factor", "Bayesian model choice". $\endgroup$
    – Xi'an
    Mar 27, 2021 at 13:39
  • 1
    $\begingroup$ beware that likelihood is not a synonym of 'probability'. in much of the question, you used the term wrong. $\endgroup$
    – carlo
    Mar 29, 2021 at 16:10

4 Answers 4

5
$\begingroup$

Before addressing the specifics of conditioning on a model or parameter, the first thing to note here is that all probability statements are conditional on implicit information. As I have noted in some other answers (e.g., here), many theories of probability regard conditional probabilities as the "primitive" in probability theory, and derive "marginal" probabilities only as a consequence of removing certain explicit conditioning events. This viewpoint is most famously associated with the axiomatic approach of the mathematician Alfréd Rényi (see e.g., Kaminski 1984). Rényi argued that every probability measure must be interpreted as being conditional on some underlying information, and that reference to marginal probabilities was merely a reference to probability where the underlying conditions are implicit.

In practical applications of probabilistic/statistical modelling, any conditioning event that holds for the entire analysis is usually removed as an explicit condition in the notation --- it is simply not useful to condition every probability statement in the analysis on the same conditioning event. Consequently, if we are working with only a single model, we would not bother to mention conditioning on the model form at all; the assumptions of the model would instead form implicit conditions for the whole analysis. Thus, explicit conditioning on the model is only useful in applications where we are considering more than one possible model form, and even then, only when the various model forms cannot fruitfully be stated as different parameter values within one overall model. This occurs in some practical modelling applications, and it also occurs when examining statistical properties of models in a meta-analytical perspective, where we remove the underlying modelling assumptions and look at statistical behaviours in their absence.

So, as a practical matter, explicit conditioning on "models" (i.e., sets of assumptions about probabilistic behaviour of observable values) is only useful when:

  • (1) There is more than one model under consideration in the analysis; and
  • (2) The models in the analysis cannot be stated more simply as different parameter ranges in a single more general model (i.e., they are not just "nested" models).

Of note here is that, under these conditions, different models will have different parameters that mean different things in the context of those models. This leads you to a problem if you want to refer to a probability like $p(M|\theta)$, which is the probability of a specific model conditional on the outcomes of a specific set of parameter values --- i.e., does the parameter $\theta$ even exist (and mean the same thing) under model $M$ and the alternative models in the analysis?

In this modelling context, in order to make conditioning probabilities of this kind make sense, you have to ensure that all parameters are well-defined regardless of which model is used. (Otherwise you may end up conditioning on parameters that don't exist.) This means that you will need to stipulate a framework where all parameters in all models exist, and you have a prior over all these parameters. Moreover, parameters that don't appear in a model don't affect that model, and so it stands to reason that your prior should treat parameters and models independently, unless we are talking about groups of parameters that may jointly exist under a single model (in which case we may allow prior dependence). The outcome of these assumptions will render conditional probabilities of models conditional on parameters trivial. We will see below that in order to make this question sensible in the context of non-nested models, we get trivial and unhelpful results.


An example: To illustrate this issue, consider an analysis where you are modelling a survival time $X \geqslant 0$ with one of two non-nested models and their corresponding parameters:

$$\begin{matrix} \text{Model } M_1 & & & X \sim \text{Ga}(\text{Shape} = 2, \text{Scale} = \theta), \quad \quad \\[6pt] \text{Model } M_2 & & & X \sim \text{Weibull}(\text{Shape} = 2, \text{Scale} = \lambda). \\[6pt] \end{matrix}$$

If you start to look at the conditional probabilities of models conditional on parameters you see an immediate problem --- the parameters for the two models are different. In order to ensure that the probabilties are well-defined (so that you can condition on them regardless of which model is used) you can stipulate that all the parameters exist under each model, but parameters don't do anything in the model if they don't appear. So in this case, we could stipulate that the parameter vector $(\theta, \lambda)$ always exists, with model $M_1$ only using the first parameter and model $M_2$ only using the second. If we do this then it obviously also makes sense that our prior distribution over the models and parameters should treat them as independent --- i.e., we have:

$$\pi(M, \theta, \lambda) = \pi(M) \pi(\theta) \pi(\lambda).$$

Using a prior of this form and applying Bayes' rule gives the trivial results:

$$\begin{align} p(M | \theta ) &= \frac{p(M, \theta)}{p(\theta) } = \frac{\pi(M) \cdot \pi(\theta)}{\pi(\theta)} = \pi(M), \\[12pt] p(M | \lambda) &= \frac{p(M, \lambda)}{p(\lambda) } = \frac{\pi(M) \cdot \pi(\lambda)}{\pi(\lambda)} = \pi(M). \\[6pt] \end{align}$$

As you can see, the prior assumptions in this case lead to the parameters giving no information on which model is used. (Note that conditioning on the observed value of $X$ will usually give information on which model is used, but that is a different question.) Consequently, the inquiry into the conditional probability of models given the true parameter values is trivial and unhelpful.


The astute reader will no doubt have noticed that this trivial outcome comes directly from prior assumptions that stipulate independence between models and parameters. Consequently, it is natural to wonder whether we might get non-trivial results if we adopt a prior that treats these as dependent. Of course, this is possible, but it doesn't seem very sensible. If the parameter $\theta$ is meaningless under model $M_2$ (and the parameter $\lambda$ is meaningless under model $M_1$) then there is no value in stipulating different prior distributions for these parameters under the two models.

$\endgroup$
6
  • $\begingroup$ Thank you, this was an excellent cover. I have/had rather different perspective that parameters are bound to the models and cannot be shared among models. The fact that two models share the parameter of the same name, type, dimension and every other attribute that I have not mentioned (i.e. relations to other parameters) may be irrelevant. This way we can only consider conditional probability of one model in condition to its parameters, since adding additional parameters has no sense (we can remove them). $\endgroup$
    – Good Luck
    Apr 3, 2021 at 11:12
  • $\begingroup$ But what is conditional probability of one model in condition to its parameters, this looks again like probability of that model. To me conditioning on parameters of a model having that parameters is just more formal way saying model has parameters. Just like an explicit form of writing. From that perspective, we only pay attention to the dimensions of measurable spaces mentioned in the comments of @Gianni. $\endgroup$
    – Good Luck
    Apr 3, 2021 at 11:17
  • $\begingroup$ Still your method to stipulate that all the parameters exist under each model cover all cases, even those I mentioned, just saying; but doesn't incorporate the point of view that parameters are per model parameters. (That view may be even wrong). Also I feel that there is no need to mention the fact that model is random variable any more, since this is obvious to me (just about everything can be represented as a random variable). I really like like your answer. $\endgroup$
    – Good Luck
    Apr 3, 2021 at 11:21
  • 1
    $\begingroup$ You can certainly take the approach that parameters only exist under their models, but if you do this then the issue becomes even more trivial, since many of the conditioning events just don't exist. $\endgroup$
    – Ben
    Apr 3, 2021 at 11:44
  • $\begingroup$ Feels great that this approach that parameters only exist under their models is possible @Ben ;) I was not so sure before. ☆☆☆☆☆ $\endgroup$
    – Good Luck
    Apr 3, 2021 at 12:14
2
+100
$\begingroup$

Short answer: for some applications, models are treated with uncertainty, but usually the interest is in $p(M | X)$, not $p(M | \theta)$.


I think the example - $p(M | \theta)$ - is obscuring the question - "Are models random variables?".[1] $p(M | \theta)$ is kinda an odd thing to ask about. Does specifying an intercept of a linear model tell you much about whether a linear or quadratic model is more plausible, independently of any data?

On the other hand, $p(M | X)$ is of intense interest in model comparison / selection tasks.

Suppose I have some data and want estimate whether a linear model $m_0$: $y = \alpha + \beta x$ is more plausible than simple quadratic model $m_1$: $y = \alpha + \beta x^2$. The set of events $M$ for the distribution $p(M)$ will then have two elements: $M = \{m_0, m_1\}$. $p(M)$ is then an assignment of values in $(0, 1)$ to each element of $M$ s.t. they sum to 1, representing their relative a priori plausibility.

If I want to determine which model is more plausible for my data, I can compute $P(M | X)$ using Bayes' theorem:

$$p(M | X) = \frac {p(X | M) p(M)}{p(X)}$$

Where

$$p(X) = \sum_{m \in M}{p(X | m)p(m)}$$

And as you have above (using $\theta_m$ to clarify that different models can have different parameters).

$$p(X | m) = \int_{\theta_m} { p(X | m, \theta_m) p(\theta_m | m)d \theta_m}$$

The posterior $p(M|X)$ allocates mass to models in $M$ that were more plausible given the data.

You might find Bayes' factors relevant here, as an example usage of posteriors over models.


[1] You'll probably see Bayesians often talking about quantifying uncertainty over sets of events/propositions/beliefs rather than "random variables" per se. Models themselves aren't random in some metaphysical sense, but we are often uncertain which model is "the best" given our observations. So it's often more natural in this context to talk about uncertainty over beliefs.


Edit: copying some notes from the comments.

To explicitly answer the question, I would say $𝑀$ is an RV as long as you interpret it carefully. If $𝐌$ is the set of all models, and $p(𝑀$) is the function that maps models to plausibilities, then $𝑀$ is an RV that just indexes our set of models $𝐌$. Basically a categorical- or function-valued RV. This is not a very common way to talk about it, though.

However, to me it's not intuitive to talk about $M$ as an RV. If somebody is about to roll a die, then it's natural to use an RV that maps the set of die faces (outcomes) to the numbers 1-6. But if somebody rolls a die and hides it, and asks what side I think is up, it's no longer "random" in the sense that there's already an outcome; I just don't know it yet. This is closer to the situation in model comparison, where it'd be more typical to talk about plausibilities than random "outcomes".


As for how a model relates to parameters. You can think of a function as a mapping from values in one set to another. Eg $𝑦=𝑚𝑥$ maps values in $\mathcal{R}^2→\mathcal{R}$. You could also say $𝑦=𝑚𝑥$ describes a set of functions, one for each possible value of $𝑚$, that map $\mathcal{R} \rightarrow \mathcal{R}$. We call $𝑚$ a parameter when we have observations for $𝑦$ and $𝑥$, but not $𝑚$. Another way to think about it is that $𝑚$ "selects" a function, such as $𝑦=2𝑥$ for $𝑚=2$. $𝑝(𝑚|𝑥,𝑦)$ is then the likelihood of param $𝑚$ for the model $𝑦=𝑚𝑥$, given our observations.

(This relates to something the other answer was trying to get at: models don't depend on parameters. For example, if we have two models: $𝑦=𝛽_0𝑥+𝛼_0$ and $𝑦=𝛾𝑥^2+𝛽_1𝑥+𝛼_1$, we can see immediately that they don't share the same parameters. If we knew the parameters for one, it wouldn't tell us anything about the plausibility of either model independent of any data.)

[...] And since models in general don't/cannot share parameters we cannot really express $𝑝(𝑀∣𝜃)$. Similar for $𝑝(𝜃∣𝑀)$ unless $𝑀$ denotes models corresponding to the parameters of those models.

I believe technically you can express both of these as long as $m \in M$ and $\theta_m$ are consistent. What's challenging is the interpretation of $p(M | \theta)$; I can't think of a use for this, and my intuition is that it's actually just $p(M)$. ($p(\theta_m | M=m)$ is fine, as it just describes the prior we have on parameters for some selected model.)

$\endgroup$
9
  • $\begingroup$ Very good read @Gianni, at this moment I cannot conclude if the models are random variables or no, even though you provided the clue that models aren't random in metaphysical sense. I would simple accept either yes or no, because so far I learned that Bayesian use random variables with $\mid$ condition. I would rather ask you in the sense of this quest, do you possible see, what I get the feeling of, what is on the right side of the $\mid$ is an attribute: say $p(\mathsf{person} \mid \mathsf{redhair})$, so the attributes limit the probability mass function of person. $\endgroup$
    – Good Luck
    Mar 30, 2021 at 23:49
  • $\begingroup$ So in $p(M | \theta)$ I get the feeling that we can ask models for the attributes, possible all models, while with $\theta$ we assume all the parameters of that model so typically we get all models by all parameters independent of data and this may be what is unusual. While if we would have just a single model then the previous probability would possible be 1, even without considering the data. Any comment on how the model relates to parameters is welcome! $\endgroup$
    – Good Luck
    Mar 30, 2021 at 23:56
  • $\begingroup$ I would say that $M$ is an RV as long as you interpret it carefully. If $\textbf{M}$ is the set of all models, and $P(M)$ is the function that maps models to plausibilities, then $M$ is an RV that just indexes our set of models $\textbf{M}$. Basically a categorical- or function-valued RV. Not a very common way to talk about it, though: en.wikipedia.org/wiki/Random_variable#Extensions $\endgroup$
    – Gianni
    Apr 1, 2021 at 1:49
  • $\begingroup$ But to me, it's not very useful or intuitive to talk about models as RVs. If somebody rolls a die, then it's natural to use an RV that maps the set of die faces to the numbers 1-6. But if somebody rolls a die and hides it, and asks what side I think is up, it's no longer "random" in the sense that there's already an outcome; I just don't know it yet. This is closer to the situation in model comparison, where it'd be more typical to talk about plausibilities than random "outcomes". $\endgroup$
    – Gianni
    Apr 1, 2021 at 1:50
  • $\begingroup$ As for how a model relates to parameters. You can think of a function as a mapping from values in one set to another. Eg $y = mx$ maps values in $\mathcal{R}^2 \rightarrow \mathcal{R}$. You could also say $y = mx$ describes a set of functions, one for each possible value of $m$, that map $\mathcal{R} \rightarrow \mathcal{R}$ We call $m$ a parameter when we have observations for $y$ and $x$, but not $m$. Another way to think about it is that $m$ "selects" a function, such as $y = 2x$ for $m=2$. $p(m | x, y)$ is then the likelihood of param $m$ for the model $y = mx$, given our observations. $\endgroup$
    – Gianni
    Apr 1, 2021 at 2:01
1
$\begingroup$

I think that the confusion you are experiencing is mainly due to notation and a bit of philosophy around bayesian probability.

  1. In general, we define a model $M$ having parameters $\theta$. This structure $M$ is "a structure", it can be a linear regression, a Tree, a Random Forest, A Neural Net, etc... $\theta$ are the parameters of the model. They are the unknowns that exists in a parameter space, and that will allow you to find the model for your specific $X$. However that definition is already somehow arbitrary, for instance we don't count hidden layer of NN or depth of Tree for Random Forest to be parameters but hyperparameters. In essence there is no difference, but we consider that "they are the model".

  2. Why do we do this trick structure/hyperparameter/parameter ? Well mainly because otherwise no problem would be solvable. If your model $M$ can be any model (you don't make a choice, but condition over all the possible structure of models), it would be an impossible problem to solve. So you set a structure $M$ and then optimize within that structure. More recently, there is an entire field of research called Neural Architecture Search or AutoML that aims at finding the best model (mainly hyperparameters) but this is a very complex task.

  3. It's important to note that the 2 formulas are actually equivalent, it is more about interpretation. The fact that every term in the first formula is conditioned on $M$ means that $M$ acts as a deterministic value. It could be a random variable but in this case, you always work under the assumption that $M$ is fully known. In the second formula, $M$ is not there because simply it is useless to add it. There is no dependence, $M$ is a feature of the world that can't be change, it is not a random variable, no need to condition on it. The way to use both formulas remains the same regardless of the interpretation.

  4. Why don't we write $P(M|\theta)$ ? In theory, one could completely compute $P(M|\theta)$, but we don't. First because it does not answer any useful question, in real life $M$ will always be specified before $\theta$. Second because as we've seen, there is no need to have a distribution over $M$, you always live in a "sub-world" where $M$ is either a fixed value of random variable or unchangeable. Finally, because that does not really make sense. Choosing a specific $M$ will specify your vector $\theta's$. For instance choosing linear regression will make $\theta's$ coefficient of the linear reg, but choosing a tree will make $\theta's$ being splits along a feature.

I hope this helps!

$\endgroup$
5
  • $\begingroup$ Nice wrap, "M it is not a random variable, no need to condition on it", this has sense and it clears the things, but possible the only argument against it is that we do use the formula with $P(\theta \mid M)$ which is what is confusing me about. And also one may say there is an infinite set of different modules possible each represented with different $\theta$s. If you ask me at this very moment I think the formula #1 is completely useless. The model is a set of parameters and no need for explicit involvement. $\endgroup$
    – Good Luck
    Mar 29, 2021 at 12:01
  • $\begingroup$ To clarify when we use the formula $P(\theta \mid M)$ this means that the $M$ is a random variable in Bayeisian tradition. Also the $\theta$ is random variable. In fact in Bayesian tradition there are no unknowns, instead we have random variables and sets of parameters, but in the upper formula all are RVs. $\endgroup$
    – Good Luck
    Mar 29, 2021 at 12:04
  • $\begingroup$ Agreed but a deterministic variable is a RV with all probability mass in a single point. So the first formula in more general, but it makes no difference in real life, because we never consider to set of all possible models, we optimize within a given set of models. $\endgroup$
    – Romain
    Mar 29, 2021 at 14:55
  • $\begingroup$ Hi, at this moment of my analysis it appears that deterministic variable is not a random variable or maybe it is a special case I don't know. $\endgroup$
    – Good Luck
    Mar 29, 2021 at 15:42
  • 3
    $\begingroup$ it can be considered as a special case. a fixed value can be described as a Dirac random value, which is saying that is not actually random. $\endgroup$
    – carlo
    Mar 29, 2021 at 16:14
0
$\begingroup$

At time of writing- Two good answers here, so I'll take a different approach and try to explain some things in "plain English."

When we talk about likelihood, think of "plausibility" rather than probability. What makes something plausible? The parameter(s) that describe the process which (assumedly) underlie the data generating process. If the parameter is a good fit, observing the data should seem plausible to you. If the parameter is a terrible fit, then observing the data should be somewhat surprising.

A "frequentist" would use calculus to optimize the parameter that makes observing the data most plausible (maximum likelihood estimation.) However, this is just a point estimate and we don't know how quickly it loses predictive accuracy if we were to nudge it.

Bayesians find this uncertainty around parameter estimates to be troublesome. And so they use Bayes Rule to flip the relationship, no longer asking, "how plausible is the data (given the parameter)?" (question 1) But rather "How plausible is the parameter (given the data)?" (question 2.)

Question 1 is often called the likelihood. Question 2 is often called the posterior.

Another element we haven't addressed is the prior. It's some belief about the parameter(s) that guides them to areas, which you believe are more reasonable. For example, if you're studying the relationship between height and weight in a linear model, you know "a priori" (pre-analysis) that there shouldn't be a negative relationship between height and weight. It would make no sense that an increase in weight should be associated with a decrease in height.

And the last element is the evidence- it's the probability of seeing the data, irrespective of any specific parameter value. In the likelihood, we talk about how plausible observing the data would be given some parameter describing the data generating process. The evidence however, marginalizes this parameter out via integration (continuous) or summation (discrete). If this sounds weird, think of it in the discrete case first. If you have data of children at a school, and more specifically you have two probabilities: (1) that a student is both (A) a member of the chess club and (B) a boy and (2) that a student is both (A) a member of the chess club and (B) a girl. Then if you marginalize out sex, you can isolate the probability of belonging to the chess club. In the continuous case, we have an infinite number of classes/buckets, which the parameter could take on.

To make things more complicated, often the most difficult part of Bayesian Inference (getting the posterior distribution) is in computing the evidence. Integrals are hard in 1D, damn tricky in 2D, extremely difficult in 3D, and 4-plus dimensions? Forget it. Fortunately, the evidence is only used to ensure that the numerator (likelihood * prior) is scaled such that it sums to 1 (a valid probability distribution.) With this in mind, we still know a good deal about what we'd like to model given just the numerator, whether or not it integrates to 1. In fact, there are several different ways by which the posterior could be approximated- MCMC sampling, grid approximation (like multidimensional Riemann sums), etc.

Anyway, all that to say, it's very often to see the posterior being "proportionate to" to the numerator. Don't let this trip you up.

The Bayesian mindset is really cool; unfortunately it's often taught in two unsatisfactory ways. (A) In a stats department with the assumption that you've had 3 semesters of calc, 1 semester of linear algebra, and perhaps even more mathematical firepower- so the proofs ought to speak for themselves. Or (B) As an anecdote or chapter in a machine learning course- these sort of books will discuss priors as little more than regularizing coefficients (not entirely false) and view Bayesian Inference as just a tool for the job, which should be used when your random forest (or insert ML model) performs poorly.

Neither of these approaches are really helpful for an outside looking in. I recommend Statistical Rethinking, by Richard McElreath. Lectures, PyMC3 code. Cheers!

$\endgroup$
1
  • $\begingroup$ Dear @jbuddy_13 is statistical plausibility term possible to formalize? $\endgroup$
    – Good Luck
    Apr 2, 2021 at 23:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.