verifying a posterior is proper

Question

There's a homework problem in a textbook that asks to verify propriety of a certain posterior distribution, and I'm having a little trouble with it. The setup is you have a logistic regression model with one predictor, and you have an improper uniform prior over $\mathbb{R}^2$.

Specifically, we assume for $i=1,\ldots,k$ that $$ y_i \mid \alpha, \beta,x_i \sim \text{Binomial}(n,\text{invlogit}(\alpha + \beta x_i)), $$ so the likelihood is $$ p(y \mid \alpha, \beta, x ) = \prod_{i=1}^k [\text{invlogit}(\alpha + \beta x_i)]^{y_i}[1-\text{invlogit}(\alpha + \beta x_i)]^{n-y_i}. $$ The trouble is that I suspect that this posterior is actually improper.

For the specific situation where $k=1$, if we use the change of variables $s_1 = \text{invlogit}(\alpha + \beta x)$ and $s_2 = \beta$, then we can see that \begin{align*} \iint_{\mathbb{R}^2}p(y \mid \alpha, \beta, x ) \text{d}\alpha \text{d}\beta &= \iint_{\mathbb{R}^2}[\text{invlogit}(\alpha + \beta x)]^{y}[1-\text{invlogit}(\alpha + \beta x)]^{n-y} \text{d}\alpha \text{d}\beta \\ &= \int_{-\infty}^{\infty}\int_0^1 s_1^{y-1}(1-s_1)^{n-y-1} \text{d}s_1 \text{d}s_2 \\ &= B(y,n-y) \int_{-\infty}^{\infty} 1 \text{d}s_2 \tag{*}\\ &= \infty. \end{align*} In the line with the asterisk we assume that $0 < y < n$, but if it doesn't, then we end up with the same thing.

Am I doing something silly here? Or is this an improper posterior?

Answer 1

For $𝑘=1$, you can see directly from your equation for the likelihood that the posterior density $𝑝(𝛼,𝛽|𝑦_1,𝑥_1)$ will be constant along parallell lines at which $𝛼+𝛽𝑥_𝑖$ takes constant values. So the posterior is indeed improper and has the shape of a ridge for $𝑘=1$. Basically, any regression line fitting the observed response at $𝑥_1$ will do equally well.

Next, suppose we have $k=2$ observations. Consider the reparameterization given by \begin{align} \eta_1 &= \alpha + \beta x_1 \\ \eta_2 &= \alpha + \beta x_2 \end{align} Since this is a linear transformation of $\alpha,\beta$ with a constant determinant the prior for $\eta_1,\eta_2$ is also uniform over $\mathbb{R}^2$, provided that $x_1\neq x_2$. Consider the further reparameterization, the inverse logit transformation \begin{align} p_i = \frac1{1+e^{-\eta_i}}, \end{align} for $i=1,2$. Clearly, $p_1,p_2$ are also a priori independent with densities given by $$ \pi(p_i)=\pi(\eta_i)\Big|\frac{d\eta_i}{dp_i}\Big|\propto \frac d{dp_i}\ln\frac{p_i}{1-p_i} = \frac1{(1-p_i)p_i} $$ These are so called improper Haldane priors, that can be interpreted as a certain form of limit of the density of a Beta distribution with both parameters approaching zero. Conditional on the data $y_1,y_2$, provided that $0<y_i<n$, the posterior marginal density for each $p_i$ are proper Beta distributions with parameters $y_i,n-y_i$. Backtransforming, the posterior distributions of $(\eta_1,\eta_2)$ and $(\alpha,\beta)$ must also be proper. This holds except in special cases such as one $y_i$ taking a value of 0 or $n$ in which case the normalising beta function $B(y_i,n-y_i)$ is infinite and the posterior of $p_i$ (and hence the posterior of $\alpha$ and $\beta$) is improper.

For $k>2$ observations, the posterior must also be also be proper since the non-normalized posterior density of $\alpha,\beta$ is bounded by the posterior based on the first $k=2$ observations.

Answer 2

It's improper, I believe. I only need to prove that $$\int\limits_{\alpha \in \mathbb{R} \\ \beta>0} p(y \mid \alpha, \beta, x) = +\infty.$$ Denote function $$\sigma = \mathrm{invlogit}$$ Now that $\sigma$ is a monotonically increasing function, when $\beta > 0$, we have $$\mathrm{\sigma}(\alpha + \beta x_i) > \mathrm{\sigma}(\alpha - \beta \max |x_i|) > 0,$$ $$1 - \mathrm{\sigma}(\alpha + \beta x_i) > 1 - \mathrm{\sigma}(\alpha + \beta \max |x_i|) > 0.$$

Thus the the integral $$\begin{aligned} \int\limits_{\alpha \in \mathbb{R} \\ \beta>0} p(y \mid \alpha, \beta, x) >& \int\limits_{\alpha \in \mathbb{R} \\ \beta>0} \prod \left[ \mathrm{\sigma}(\alpha - \beta \max |x_i|) \right]^{y_i} \left[ 1 - \mathrm{\sigma}(\alpha + \beta \max |x_i|) \right]^{n_i - y_i} \\ >& \int\limits_{\alpha \in \mathbb{R} \\ \beta>0} \prod \left[ \mathrm{\sigma}(\alpha - \beta \max |x_i|) \right]^{\max n_i} \left[ 1 - \mathrm{\sigma}(\alpha + \beta \max |x_i|) \right]^{\max n_i} \\ >& \int\limits_{\alpha \in \mathbb{R} \\ \beta>0} \left[ \mathrm{\sigma}(\alpha - \beta \max |x_i|) \right]^{k\max n_i} \left[ 1 - \mathrm{\sigma}(\alpha + \beta \max |x_i|) \right]^{k\max n_i} \\ \end{aligned}$$

More properties about $\sigma$ are needed: $$\left( \sigma(x) \right)^N = \frac{1}{(1+e^{-x})^N} > \dfrac{1}{2^N (\max\{1,e^{-x}\}) ^N} = \dfrac{1}{2^N (\max\{1,e^{-Nx}\})} > \dfrac{1}{2^N}\sigma(Nx)$$

Let $\xi = \alpha - \beta \max |x_i|$, $\eta = \alpha + \beta \max |x_i|, N=k\max n_i$, then \begin{aligned} \int\limits_{\alpha \in \mathbb{R} \\ \beta>0} p(y \mid \alpha, \beta, x) >& \int\limits_{\alpha \in \mathbb{R} \\ \beta>0} \left[ \mathrm{\sigma}(\alpha - \beta \max |x_i|) \right]^{N} \left[ 1 - \mathrm{\sigma}(\alpha + \beta \max |x_i|) \right]^{N} \\ \propto& \int\limits_{-\infty < \xi < \eta < +\infty} \left[ \mathrm{\sigma}(\xi) \right]^{N} \left[ \mathrm{\sigma}(-\eta) \right]^{N}\\ >& \frac{1}{2^{2N}}\int\limits_{\xi}^{+\infty} \Big( \int\limits_{-\infty}^{+\infty} \mathrm{\sigma}(N\xi) \mathrm{d}\xi \Big) ~ \mathrm{\sigma}(- N\eta) \mathrm{d}\eta \\ =& +\infty \end{aligned}

Answer 3

I've already accepted an answer, but I did want to point out that the posterior isn't proper for all possible data sets. The posterior is proportional to the likelihood, which is $$\prod_{i=1}^k [\text{invlogit}(\alpha + \beta x_i)]^{y_i}[1-\text{invlogit}(\alpha + \beta x_i)]^{n-y_i}. $$ If $y_1 = y_2 = \cdots = y_k = n$, then this simplifies to $$ \prod_{i=1}^k [\text{invlogit}(\alpha + \beta x_i)]^n, $$

and we can see that \begin{align*} &\int_{-\infty}^{\infty}\int_{-\infty}^{\infty} \prod_{i=1}^k [\text{invlogit}(\alpha + \beta x_i)]^n \text{d}\alpha \text{d}\beta\\ &\ge \int_0^{\infty}\int_{-\infty}^{\infty} \prod_{i=1}^k [\text{invlogit}(\alpha + \beta x_i)]^n \text{d}\alpha \text{d}\beta \\ &\ge \int_0^{\infty}\int_{-\infty}^{\infty} [\text{invlogit}(\alpha + \beta x_{(1)})]^{nk} \text{d}\alpha \text{d}\beta \\ &\ge \int_0^{\infty}\int_{-\infty}^{\infty} [\text{invlogit}(r_1)]^{nk} \text{d}r_1 \text{d}r_2 \\ &= \infty. \end{align*}