7
$\begingroup$

There's a homework problem in a textbook that asks to verify propriety of a certain posterior distribution, and I'm having a little trouble with it. The setup is you have a logistic regression model with one predictor, and you have an improper uniform prior over $\mathbb{R}^2$.

Specifically, we assume for $i=1,\ldots,k$ that $$ y_i \mid \alpha, \beta,x_i \sim \text{Binomial}(n,\text{invlogit}(\alpha + \beta x_i)), $$ so the likelihood is $$ p(y \mid \alpha, \beta, x ) = \prod_{i=1}^k [\text{invlogit}(\alpha + \beta x_i)]^{y_i}[1-\text{invlogit}(\alpha + \beta x_i)]^{n-y_i}. $$ The trouble is that I suspect that this posterior is actually improper.

For the specific situation where $k=1$, if we use the change of variables $s_1 = \text{invlogit}(\alpha + \beta x)$ and $s_2 = \beta$, then we can see that \begin{align*} \iint_{\mathbb{R}^2}p(y \mid \alpha, \beta, x ) \text{d}\alpha \text{d}\beta &= \iint_{\mathbb{R}^2}[\text{invlogit}(\alpha + \beta x)]^{y}[1-\text{invlogit}(\alpha + \beta x)]^{n-y} \text{d}\alpha \text{d}\beta \\ &= \int_{-\infty}^{\infty}\int_0^1 s_1^{y-1}(1-s_1)^{n-y-1} \text{d}s_1 \text{d}s_2 \\ &= B(y,n-y) \int_{-\infty}^{\infty} 1 \text{d}s_2 \tag{*}\\ &= \infty. \end{align*} In the line with the asterisk we assume that $0 < y < n$, but if it doesn't, then we end up with the same thing.

Am I doing something silly here? Or is this an improper posterior?

$\endgroup$
  • 2
    $\begingroup$ At a glance--without considering the details--I find it surpassingly strange that (a) you integrate over both positive and negative values of $\beta$ and (b) a factor of $1/\beta$ does not appear when you change variables. Perhaps, then, a little attention to the mechanics of integration will solve your problem. $\endgroup$ – whuber Feb 6 at 0:20
  • $\begingroup$ I haven’t done any math to back this up, but my both my intuition and my memory say you are right and that the posterior need not be proper. By analogy, if you fix $\beta = 0$, a flat prior on $\alpha$ is the Haldane prior, which does not always lead to proper posteriors. $\endgroup$ – guy Feb 6 at 0:30
  • 2
    $\begingroup$ For $k=1$, you can see directly from your equation for the likelihood that the posterior density $p(\alpha,\beta|y_1,x_1)$ will be constant along parameter combinations for which $\alpha + \beta x_i$ takes constant values. So the posterior is indeed improper and has the shape of a ridge for $k=1$. Basically, any regression line fitting the observed response at $x_1$ will fit the data. But for $k>1$, I would be surprised if the posterior is not proper except in degenerate cases such as $x_1=x_2$ or cases of linear separation. $\endgroup$ – Jarle Tufto Feb 6 at 16:35
  • 1
    $\begingroup$ @Xi'an that's where the $-1$s came from $\endgroup$ – Taylor Feb 7 at 14:46
  • 3
    $\begingroup$ The explanation by @JarleTufto is completely spot-on: the distribution of $Y$ only depends on $\alpha+\beta x$, hence cannot bring information about $\alpha$ and $\beta$. Hence an improper posterior. There is also an issue for more observations if all $y_i$'s are all equal to $0$ or all equal to $n$. $\endgroup$ – Xi'an Feb 7 at 15:24
3
$\begingroup$

For $𝑘=1$, you can see directly from your equation for the likelihood that the posterior density $𝑝(𝛼,𝛽|𝑦_1,𝑥_1)$ will be constant along parallell lines at which $𝛼+𝛽𝑥_𝑖$ takes constant values. So the posterior is indeed improper and has the shape of a ridge for $𝑘=1$. Basically, any regression line fitting the observed response at $𝑥_1$ will do equally well.

Next, suppose we have $k=2$ observations. Consider the reparameterization given by \begin{align} \eta_1 &= \alpha + \beta x_1 \\ \eta_2 &= \alpha + \beta x_2 \end{align} Since this is a linear transformation of $\alpha,\beta$ with a constant determinant the prior for $\eta_1,\eta_2$ is also uniform over $\mathbb{R}^2$, provided that $x_1\neq x_2$. Consider the further reparameterization, the inverse logit transformation \begin{align} p_i = \frac1{1+e^{-\eta_i}}, \end{align} for $i=1,2$. Clearly, $p_1,p_2$ are also a priori independent with densities given by $$ \pi(p_i)=\pi(\eta_i)\Big|\frac{d\eta_i}{dp_i}\Big|\propto \frac d{dp_i}\ln\frac{p_i}{1-p_i} = \frac1{(1-p_i)p_i} $$ These are so called improper Haldane priors, that can be interpreted as a certain form of limit of the density of a Beta distribution with both parameters approaching zero. Conditional on the data $y_1,y_2$, provided that $0<y_i<n$, the posterior marginal density for each $p_i$ are proper Beta distributions with parameters $y_i,n-y_i$. Backtransforming, the posterior distributions of $(\eta_1,\eta_2)$ and $(\alpha,\beta)$ must also be proper. This holds except in special cases such as one $y_i$ taking a value of 0 or $n$ in which case the normalising beta function $B(y_i,n-y_i)$ is infinite and the posterior of $p_i$ (and hence the posterior of $\alpha$ and $\beta$) is improper.

For $k>2$ observations, the posterior must also be also be proper since the non-normalized posterior density of $\alpha,\beta$ is bounded by the posterior based on the first $k=2$ observations.

$\endgroup$
  • $\begingroup$ I'm not sure our two answers are in disagreement mathematically. I'm saying the integral is infinite when each $y_i=n$, and you're saying the integral is finite whenever all $y_i$ are between the endpoints strictly. Also, this might be a thing that depends on who you ask, but I was under the impression that if a posterior is not proper for all possible data points, then it is defined to be "improper." Haldane's prior is an example where this happens. $\endgroup$ – Taylor Feb 14 at 18:47
  • $\begingroup$ But your inequalities says that the integral of the unnormalised posterior (for any $y_i$) (the left hand side of the first inequality) is infinite so there is a disagreement. I'm not sure but the last step combining the two integrals seems to involve not only what you have defined as $s_1$ and $s_2$ but also $\mbox{invlogit}(\alpha + \beta x_{(n)})$ etc. so maybe that's were the error is. $\endgroup$ – Jarle Tufto Feb 14 at 19:25
  • 1
    $\begingroup$ yes you're right. The integrands are different on those two places so you can't combine them. $\endgroup$ – Taylor Feb 16 at 19:29
  • $\begingroup$ Unfortunately, this assumes $n>1$ which is the most common case. $\endgroup$ – Taylor Feb 17 at 16:03
2
$\begingroup$

It's improper, I believe. I only need to prove that $$\int\limits_{\alpha \in \mathbb{R} \\ \beta>0} p(y \mid \alpha, \beta, x) = +\infty.$$ Denote function $$\sigma = \mathrm{invlogit}$$ Now that $\sigma$ is a monotonically increasing function, when $\beta > 0$, we have $$\mathrm{\sigma}(\alpha + \beta x_i) > \mathrm{\sigma}(\alpha - \beta \max |x_i|) > 0,$$ $$1 - \mathrm{\sigma}(\alpha + \beta x_i) > 1 - \mathrm{\sigma}(\alpha + \beta \max |x_i|) > 0.$$

Thus the the integral $$\begin{aligned} \int\limits_{\alpha \in \mathbb{R} \\ \beta>0} p(y \mid \alpha, \beta, x) >& \int\limits_{\alpha \in \mathbb{R} \\ \beta>0} \prod \left[ \mathrm{\sigma}(\alpha - \beta \max |x_i|) \right]^{y_i} \left[ 1 - \mathrm{\sigma}(\alpha + \beta \max |x_i|) \right]^{n_i - y_i} \\ >& \int\limits_{\alpha \in \mathbb{R} \\ \beta>0} \prod \left[ \mathrm{\sigma}(\alpha - \beta \max |x_i|) \right]^{\max n_i} \left[ 1 - \mathrm{\sigma}(\alpha + \beta \max |x_i|) \right]^{\max n_i} \\ >& \int\limits_{\alpha \in \mathbb{R} \\ \beta>0} \left[ \mathrm{\sigma}(\alpha - \beta \max |x_i|) \right]^{k\max n_i} \left[ 1 - \mathrm{\sigma}(\alpha + \beta \max |x_i|) \right]^{k\max n_i} \\ \end{aligned}$$

More properties about $\sigma$ are needed: $$\left( \sigma(x) \right)^N = \frac{1}{(1+e^{-x})^N} > \dfrac{1}{2^N (\max\{1,e^{-x}\}) ^N} = \dfrac{1}{2^N (\max\{1,e^{-Nx}\})} > \dfrac{1}{2^N}\sigma(Nx)$$

Let $\xi = \alpha - \beta \max |x_i|$, $\eta = \alpha + \beta \max |x_i|, N=k\max n_i$, then \begin{aligned} \int\limits_{\alpha \in \mathbb{R} \\ \beta>0} p(y \mid \alpha, \beta, x) >& \int\limits_{\alpha \in \mathbb{R} \\ \beta>0} \left[ \mathrm{\sigma}(\alpha - \beta \max |x_i|) \right]^{N} \left[ 1 - \mathrm{\sigma}(\alpha + \beta \max |x_i|) \right]^{N} \\ \propto& \int\limits_{-\infty < \xi < \eta < +\infty} \left[ \mathrm{\sigma}(\xi) \right]^{N} \left[ \mathrm{\sigma}(-\eta) \right]^{N}\\ >& \frac{1}{2^{2N}}\int\limits_{\xi}^{+\infty} \Big( \int\limits_{-\infty}^{+\infty} \mathrm{\sigma}(N\xi) \mathrm{d}\xi \Big) ~ \mathrm{\sigma}(- N\eta) \mathrm{d}\eta \\ =& +\infty \end{aligned}

$\endgroup$
  • 1
    $\begingroup$ I've made some edition. Hope it's right. $\endgroup$ – Silly Song Feb 27 at 0:11
  • 1
    $\begingroup$ In the second last step, I think have the limits of the double integral are wrong. I instead get \begin{align} \int_{-\infty}^\infty\sigma(-N\eta)\int_{-\infty}^\eta\sigma(N\xi) d\xi d\eta \end{align} Using Maple, I find that this double integral is finite and equals $\pi/(6N^2)$. So all you can say based on your derivation is that normalising constant of the posterior of $\alpha,\beta$ is larger than something finite. $\endgroup$ – Jarle Tufto Mar 1 at 17:36
1
$\begingroup$

I've already accepted an answer, but I did want to point out that the posterior isn't proper for all possible data sets. The posterior is proportional to the likelihood, which is $$\prod_{i=1}^k [\text{invlogit}(\alpha + \beta x_i)]^{y_i}[1-\text{invlogit}(\alpha + \beta x_i)]^{n-y_i}. $$ If $y_1 = y_2 = \cdots = y_k = n$, then this simplifies to $$ \prod_{i=1}^k [\text{invlogit}(\alpha + \beta x_i)]^n, $$

and we can see that \begin{align*} &\int_{-\infty}^{\infty}\int_{-\infty}^{\infty} \prod_{i=1}^k [\text{invlogit}(\alpha + \beta x_i)]^n \text{d}\alpha \text{d}\beta\\ &\ge \int_0^{\infty}\int_{-\infty}^{\infty} \prod_{i=1}^k [\text{invlogit}(\alpha + \beta x_i)]^n \text{d}\alpha \text{d}\beta \\ &\ge \int_0^{\infty}\int_{-\infty}^{\infty} [\text{invlogit}(\alpha + \beta x_{(1)})]^{nk} \text{d}\alpha \text{d}\beta \\ &\ge \int_0^{\infty}\int_{-\infty}^{\infty} [\text{invlogit}(r_1)]^{nk} \text{d}r_1 \text{d}r_2 \\ &= \infty. \end{align*}

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.