22
$\begingroup$

My question is: What is the mathematical relationship between the Beta distribution and the coefficients of the logistic regression model?

To illustrate: the logistic (sigmoid) function is given by

$$f(x) = \frac{1}{1+\exp(-x)}$$

and it is used to model probabilities in the logistic regression model. Let $A$ be a dichotomous $(0,1)$ scored outcome and $X$ a design matrix. The logistic regression model is given by

$$P(A=1|X) = f(X \beta).$$

Note $X$ has a first column of constant $1$ (intercept) and $\beta$ is a column vector of regression coefficients. For example, when we have one (standard-normal) regressor $x$ and choose $\beta_0=1$ (intercept) and $\beta_1=1$, we can simulate the resulting 'distribution of probabilities'.

Histogram of P(A=1|X)

This plot reminds of the Beta distribution (as do plots for other choices of $\beta$) whose density is given by

$$g(y;p,q) = \frac{\Gamma(p)\Gamma(q)}{\Gamma(p+q)} y^{(p-1)} (1-y)^{(q-1)}.$$

Using maximum likelihood or methods of moments it is possible to estimate $p$ and $q$ from the distribution of $P(A=1|X)$. Thus, my question comes down to: what is the relationship between choices of $\beta$ and $p$ and $q$? This, to begin with, adresses the bivariate case given above.

$\endgroup$
1
  • $\begingroup$ I was just wondering this 3 hours ago in my Bayesian statistics class $\endgroup$
    – Alchemist
    Jan 31, 2017 at 23:44

3 Answers 3

20
$\begingroup$

Beta is a distribution of values in $(0,1)$ range that is very flexible in it's shape, so for almost any unimodal empirical distribution of values in $(0,1)$ you can easily find parameters of such beta distribution that "resembles" shape of the distribution.

Notice that logistic regression provides you with conditional probabilities $\Pr(Y=1\mid X)$, while on your plot you are presenting us the marginal distribution of predicted probabilities. Those are two different things to talk about.

There is no direct relation between logistic regression parameters and parameters of beta distribution when looking on the distribution of predictions from logistic regression model. Below you can see data simulated using normal, exponential and uniform distributions transformed using logistic function. Besides using exactly the same parameters of logistic regression (i.e. $\beta_0 = 0, \beta_1 = 1$), the distributions of predicted probabilities are very different. So distribution of predicted probabilities depends not only on parameters of logistic regression, but also on distributions of $X$'s and there is no simple relation between them.

Logistic function of data simulated under normal, exponential and uniform distributions

Since beta is a distribution of values in $(0,1)$, then it cannot be used to model binary data as logistic regression does. It can be used to model probabilities, in such way we use beta regression (see also here and here). So if you are interested as the probabilities (understood as random variable) behave, you can use beta regression for such purpose.

$\endgroup$
6
  • $\begingroup$ So if Beta can approximate any such distribution, shouldn't there be a relationship between its parameters and $\beta$? $\endgroup$
    – tomka
    Jan 31, 2017 at 14:57
  • 4
    $\begingroup$ @tomka but the distribution depends on distribution of your data and on the parameters, so even is such relationship exists it's a very complicated one. There is obviously no direct relationship between regression parameters and parameters of beta distribution. Try simulating logistic regression predictions under the same parameters using different distributions for $X$, the marginal distribution will differ in each case. $\endgroup$
    – Tim
    Jan 31, 2017 at 15:01
  • 4
    $\begingroup$ The beta distribution is not that flexible -- it cannot approximate multimodal distributions. $\endgroup$
    – Marcus P S
    Jan 31, 2017 at 17:10
  • $\begingroup$ @MarcusPS I made it more clear. $\endgroup$
    – Tim
    Jan 31, 2017 at 19:12
  • 2
    $\begingroup$ @MarcusPS except the special case of multimodal distributions with modes at 0 and 1 ... $\endgroup$
    – Ben Bolker
    Feb 1, 2017 at 2:13
5
$\begingroup$

Logistic regression is a special case of a Generalized Linear Model (GLM). In this particular case of binary data, the logistic function is the canonical link function that transforms the non-linear regression problem at hand into a linear problem. GLMs are somewhat special, in the sense that they apply only to distributions in the exponential family (such as the Binomial distribution).

In Bayesian estimation, the Beta distribution is the conjugate prior to the binomial distribution, which means that a Bayesian update to a Beta prior, with binomial observations, will result in a Beta posterior. So if you have counts for observations of binary data, you can get an analytical Bayesian estimate of the parameters of the binomial distribution by using a Beta prior.

So, along the lines of what has been said by other, I don't think there is a direct relation, but both the Beta distribution and logistic regression have close relationships with estimating the parameters of something that follows a binomial distribution.

$\endgroup$
1
  • 1
    $\begingroup$ I already +1'd for mentioning Bayesian perspective, but notice that in case of regression model we do not use beta-binomial model and beta distribution in general is not used as a prior for parameters -- at least in case of typical Bayesian logistic regression. So this does not directly translate to beta-binomial model. $\endgroup$
    – Tim
    Jan 31, 2017 at 21:58
4
$\begingroup$

Maybe there is no direct connection? The distribution of $P(A=1|X)$ largely depends on your simulation of $X$. If you simulated $X$ with $N(0,1)$, $\exp(-X\beta)$ will have log-normal distribution with $\mu=-1$ given $\beta_0=\beta_1=1$. The distribution of $P(A=1|X)$ can then be found explicitly: with c.d.f. $$F(x)=1-\Phi\left[\ln\left(\frac{1}{x}-1\right)+1\right],$$ inverse c.d.f. $$Q(x)=\frac{1}{1+\exp(\Phi^{-1}(1-x)-1)},$$ and p.d.f. $$f(x)=\frac{1}{x(1 - x)\sqrt{2\pi}}\exp\left(-\frac{(\ln(1/x-1)+1)^2}{2}\right),$$ which do not resemble those of Beta distribution.

You can verify the results given above in R:

n = 100000

X = cbind(rep(1, n), rnorm(n)) # simulate design matrix
Y = 1 / (exp(-X %*% c(1,1)) + 1) # P(A=1|X)

Z1 = 1 / (rlnorm(n, -1, 1) + 1) # simulate from lognormal directly
Z2 = 1 / (1 + exp(qnorm(runif(n)) - 1)) # simulate with inverse CDF

# Kolmogorov–Smirnov test
ks.test(Y, Z1)
ks.test(Y, Z2)

# plot fitted density
new.pdf = function(x) {
  1 / (x * (1 - x) * sqrt(2 * pi)) * exp(-0.5 * (log(1 / x - 1) + 1)^2)
}
hist(Y, breaks = "FD", probability = T)
curve(new.pdf, col = 4, add = T)

enter image description here

$\endgroup$
4
  • $\begingroup$ My $x$ is indeed standard-normal (I made an edit). Your density $f(x)$ has support over $[-\inf, \inf]$, whereas the density of $P(A|X)$ should have support only on $[0,1]$. In fact your $f(x)$ should be the standard normal. In other words you have not yet shown the distribution of $P(A|X)$. $\endgroup$
    – tomka
    Jan 31, 2017 at 14:34
  • $\begingroup$ @tomka Logarithm put $1/x-1>0$, so $x\in(0,1)$. Also $f$ is not pdf of standard normal, note the denominator. $\endgroup$
    – Francis
    Jan 31, 2017 at 14:38
  • $\begingroup$ Why would the CLT have any applicability to the distribution of a regressor variable $X$?? $\endgroup$
    – whuber
    Jan 31, 2017 at 19:32
  • $\begingroup$ @whuber: looks like I have mistaken something, I removed that part. $\endgroup$
    – Francis
    Jan 31, 2017 at 20:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.