Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

The snippet from the book is pasted below. I don't understand how the denominator is not a function of theta as it is integrated over theta

The snippet from the book is pasted above. I don't understand how the denominator is not a function of theta as it is integrated over theta. Thank you.

share|improve this question
    
Because it's a definite integral. –  Glen_b Jun 16 at 8:44

3 Answers 3

When you calculate a definite integral over some variable, and the limits don't contain that variable, the variable doesn't appear in the value. Indeed, you can substitute a different variable (integrate w.r.t. a different dummy) while leaving the integral unchanged.

So

$$\int_{-\infty}^{\infty} g(\theta) d\theta$$

isn't a function of $\theta$.


Edit - since more explanation is required (I am trying to convey an intuitive understanding, so this is a bit handwavy):

Note that when you do an indefinite integral, $\int g(x) dx$, the variable can appear in the result: $G(x)+c$. But when you do a definite integral $\int_a^b g(x) dx$, you substitute the limits into the antiderivative $(G(x)+c)|_a^b$, so every place the variable appeared in the antiderivative is replaced by the limits: $G(b)-G(a)$, leaving the variable absent from the result.

[This is not restricted to linear functions!]

share|improve this answer
    
I'm rusty on my integrals. Is this only true if the function is linear in $\theta$? –  jabberwocky Jun 16 at 9:07
3  
sometimes its good to just provide a simple concrete example: What is $\int_0^5 x^2 dx$? What is $\int_0^\pi \sin{x}dx$? Are they functions of $x$? –  Benjamin Jun 16 at 11:30

One source of confusion here may be that $\theta$ is used to denote two different things in the equation. $\theta$ appearing in the numerator is the same $\theta$ as in the left-hand-side, namely, the particular parameter values whose posterior density is obtained by substituting that particular $\theta$ on the right-hand-side.

Meanwhile, the $\theta$ in the denominator is just the variable of integration. As pointed out by @Glen_b, a definite integral is not a function of the variable of integration. Especially, \begin{equation} \int f(\alpha) d\alpha = \int f(\theta) d\theta , \end{equation} as long as the limits (the area over which the integral is computed) are the same. It might be beneficial to write Bayes' rule using a different symbol in the denominator, but I guess the same symbol is used to denote that both refer to the same quantity, i.e., the same parameters of the model.

An example

Consider tossing a (possibly biased) coin once, and observing heads. Let our parameter $\theta$ be the probability that the coin comes up heads on a single toss. Define prior (before the toss) as uniform \begin{equation} p(\theta) = 1,~\theta \in [0,1]. \end{equation} Now, by definition of the parameter (and our data: heads once), \begin{equation} p(D\mid \theta) = p(\mathrm{heads} \mid \theta) = \theta \end{equation} Thus, the numerator is $p(\theta)p(D \mid \theta) = \theta$ when $\theta \in [0,1]$. The denominator is \begin{equation} \int_0^1 p(\theta) p(\mathrm{heads} \mid \theta) d\theta = \int_0^1 \theta d\theta = \left. \frac{1}{2}\theta^2 \right|_{\theta=0}^{\theta = 1} = \frac{1}{2}. \end{equation} As expected, this is not a function of $\theta$. It would be a function of the data $D$, but here for simplicity I have considered only a fixed $D$. Note that we could have denoted the parameter by any other symbol while computing the denominator: \begin{equation} \int_0^1 p(\alpha) p(\mathrm{heads} \mid \alpha) d\alpha = \int_0^1 \alpha d\alpha = \left. \frac{1}{2}\alpha^2 \right|_{\alpha=0}^{\alpha = 1} = \frac{1}{2}. \end{equation} The posterior is then \begin{equation} p(\theta \mid D=\mathrm{heads}) = \frac{p(\theta)p(\mathrm{heads} \mid \theta)}{\int_0^1 p(\theta) p(\mathrm{heads} \mid \theta) d\theta} = \frac{\theta}{\frac{1}{2}} = 2\theta, \end{equation} which is indeed proportional to $p(\theta)p(\mathrm{heads} \mid \theta)=\theta$.

share|improve this answer

The denominator in 11.1 is a function of $\theta$. However the denominator takes the same value irrespective of of which particular value of $\theta$ we are interested in because we intergrate over the whole range of possible values for $\theta$. The equation below 11.1 states that $P(\theta|D)$ is proportional to $P(\theta)L(\theta;D)$, not equal to it. i.e. $$P(\theta|D) = kP(\theta)P(D|\theta)$$ Here, the constant of proportionality, $k$ is the denominator form 11.1: $$ k= \frac{1}{\int{P(\theta)P(D|\theta)d\theta}} $$ We can use this trick because often we are interested either in finding the $\theta$ that maximizes $P(\theta|D)$ rather than the value of $P(\theta|D)$ and so the constant doesn't matter, or we are interested in the ratio of $P(\theta|D)$ for two different values of theta, and so the $k$s cancel out.

This is a useful trick because the denominator can be difficult or impossible to calculate in many cases.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.