5
$\begingroup$

Supposing that there is a binomial distribution ${\rm Bin}(m|N, \mu)$, I think usually $N$ and $\mu$ are parameters and not random variables (or events), thus the notation here ${\rm Bin}(m|N, \mu)$ seems somewhat misleading. Here, the binomial distribution ${\rm Bin}(m|N, \mu)$ is \begin{align*} {\rm Bin}(m|N, \mu) = \begin{pmatrix} N \\ m\end{pmatrix} \mu^m (1 - \mu)^{N-m} \end{align*} in the PRML notation.

My Question: However, is this notation really misleading? If not, what is the definition of $p(N)$ here?

If we would like to consider the Bayesian inference, we often introduce a beta distributions ${\rm Beta}(\mu | a, b)$ as the conjugate prior distribution of the parameter $\mu$.

Then, the PRML eq. (2.17) says that the posterior distribution of $p$ is given by

\begin{align*} p(\mu|m, N, a, b) \propto {\rm Bin}(m|N, \mu) {\rm Beta}(\mu | a, b) \end{align*}

via the Bayes' theorem and I would like to show it explicitly. The $\propto$ means that if we ignore all factors independent of $\mu$ we obtain this expression.

To show this, at first I assumed $m$ and $a, b$ are conditionally independent given $\mu$ and $N$.Thus \begin{align*} p(m|N, \mu) = p(m|N, \mu, a, b) &= \frac{p(m, N, \mu, a, b)}{p(N, \mu, a, b)} = \frac{p(m, N|\mu, a, b) p(\mu | a, b) p(a, b)}{p(N|\mu) p(\mu | a, b)} \\ &= \frac{p(m, N|\mu, a, b)p(a, b)}{p(N|\mu)}. \end{align*} Then we get \begin{align*} p(m|N, \mu) p(\mu | a, b) &= \frac{p(m, N|\mu, a, b)p(a, b)}{p(N | \mu)} \frac{p(\mu, a, b)}{p(a, b)} \\ &= \frac{p(\mu, m, N, a, b)}{p(N|\mu)}\\ &= \frac{p(\mu|m, N, a, b)p(m, N, a, b)}{p(N|\mu)}. \end{align*} Obviously, the factor $p(N|\mu)^{-1}$ appears and contradicts the above statement.

Therefore I believe that $N$ (and hyperparameters $a, b$) are parameters, not random variables and thus we don't have to consider probabilities like $p(N)$ or $p(N|\mu)$.

Am I right in my thinking? If so, it would be even more appreciated if you could show the correct normalized expression with parameters separated.

$\endgroup$
4
  • $\begingroup$ I don't quite get what you are asking, but $N, \mu$ are for sure parameters that you have to fix a priori... having said that, you might have prior knowledge on what value they can assume (maybe you know that $\mu$ should tend to, for example, 0.7, and thus you add a prior over it's distribution, but this is meant to find the best distribution that fits your data) $\endgroup$
    – Alberto
    Jul 16, 2022 at 11:01
  • $\begingroup$ @AlbertoSinigaglia Thank you. So it is true that $N, \mu$ are parameters... Then, does assuming the prior distribution of $\mu$ like above implies promoting $\mu$ from a parameter to a random variable? I think so, but I'm not confident. I think $N$ is still a parameter and $p(N)$ is meaningless. Is there any better way to denote the difference between those a priori parameters and random variables? I think it is better to denote $Bin(m | \mu, N)$ by $Bin(m ; \mu, N)$ or $Bin_{\mu, N}(m)$. If we use the pipe "|" I can't ditsinguish parameters from random variables. Is my opinion correct? $\endgroup$
    – Keyflux
    Jul 16, 2022 at 11:11
  • 1
    $\begingroup$ yes, however, you can think it in more naive way saying that $\mu$ is a RV, however your binomial treats it as a parameter, in a sense that you first get a sample of $\mu$ from it's distribution (random variable), and after that, you fix that as value, and you estimate your binomial distribution... in other words, you are conditioning over $\mu$ (which is the $"|"$ in your notation) $\endgroup$
    – Alberto
    Jul 16, 2022 at 12:51
  • $\begingroup$ There are situations where N is treated as an unknown constant to be estimated, see stats.stackexchange.com/questions/123367/… $\endgroup$ Jul 16, 2022 at 15:26

2 Answers 2

6
$\begingroup$
  1. One can well think of situations in which $N$ is random, even with a frequentist interpretation (there may be a sequence of experiments with different $N$ that can be interpreted as varying randomly). For example, a service center may be interested in what percentage of calls they receive is about a certain product. They may collect data from one or several hours over the day. Number of calls in an hour $N$ can be modelled by a Poisson distribution, and the percentage of interest can then be estimated conditionally, but note that the distribution of $N$ may have impact on the sampling distribution of the estimator and, in advance, on data collection planning.

  2. The use of the term "parameter" doesn't necessarily imply that a quantity is not random (quite obviously not in Bayesian analysis, but neither necessarily in a frequentist approach, where in some situations parameters can be randomly drawn, see above).

  3. Of course, in the most standard frequentist analyses, parameters are in fact fixed and nonrandom, and many frequentists then would think that writing down distributions using "|" as if they were conditional on something random is misleading and wrong (unless of course you either do a Bayesian analysis or there is a frequentist mechanism behind the parameter). Personally I think to some extent this is a matter of taste. I see why people think it's wrong, however in many places it can smoothly be used without leading to any trouble. If you want, it offers a hand to the Bayesian who may want to model the parameters as random even if the frequentist thinks they are not.

  4. Even if you do a Bayesian analysis, in fact you could treat the $N$ as fixed if there is no uncertainty about it (I expect this to happen in the majority of situations, but not in all of them). In other situations (see above) there may be uncertainty about it, and then obviously the $p(N)$ has to be defined expressing this uncertainty appropriately. For example it could be a Poisson in the situation 1 above, potentially with a hyperprior on top for the $\lambda$ parameter of the Poisson. In any case there is no fixed choice for this, it depends on the situation and how uncertainty there plays out.

$\endgroup$
1
  • $\begingroup$ Thank you for the clear explanation! Both answers are very helpful and it is hard to choose an accepted answer, but I choose this one because of the extensive background explanation. $\endgroup$
    – Keyflux
    Jul 19, 2022 at 0:41
8
$\begingroup$

Binomial distribution has two parameters: sample size and probability of success. The parameters can either be fixed or be random variables, can be known or need to be estimated. If they are to be estimated, then depending on your preferred approach they can be treated as random variables (Bayesian school of thought) or not (frequentist or likelihoodist). So both cases are possible and you need to consult the book about what exactly the author says.

That said, using conditional “$| $” (vs “$; $) notation often, but not necessary, suggests that the author treats them as random variables. If they talk about posterior, that this is clearly the case, since there are no non-Bayesian posteriors.

$\endgroup$
2
  • 1
    $\begingroup$ +1: The bit about notation is priceless; i haven't seen that explained anywhere else. $\endgroup$ Jul 16, 2022 at 12:22
  • $\begingroup$ Thank you for the clear explanation! $\endgroup$
    – Keyflux
    Jul 19, 2022 at 0:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.