19
$\begingroup$

I'm trying to understand how logistic regression uses the binomial distribution.

Let's say I'm studying nest success in birds. The probability of a nest being successful is 0.6. Using the binomial distribution, I can calculate the probability of r successes given n trials (numbers of nests studied).

But how is the binomial distribution used in a modelling context? Let's say I want to know how mean daily temperature affects nest success and I use logistic regression to explore this question.

Within the context I've described, how does the logistic regression use the binomial distribution?

I'm looking for an intuitive answer, therefore an answer without equations! I think equations are only useful once understanding has been achieved at an intuitive level.

$\endgroup$
  • $\begingroup$ possible duplicate of Help Translating R GLM Command to Math Notation $\endgroup$ – Sycorax says Reinstate Monica Mar 26 '14 at 18:34
  • 2
    $\begingroup$ That's a good thread to reference, @user777, but I think it may not work as a duplicate if the OP specifically requests an explanation without equations. A good question, though, is whether the role of the binomial distribution can be well explained w/o any equations. Certainly LR can be given an intuitive explanation, but the role of the binomial w/i LR is inherently somewhat mathematical. $\endgroup$ – gung - Reinstate Monica Mar 26 '14 at 18:50
  • $\begingroup$ @gung Apologies to luciano. Seeing "binomial distribution" overwrote "no equations" when I considered possible answers. Clearly that was inconsistent with OP's request. $\endgroup$ – Sycorax says Reinstate Monica Mar 26 '14 at 19:00
  • $\begingroup$ I feel I've grasped the idea of using the log odds scale for the regression then converting from the logit scale to the 0-1 scale. But I keep reading things along the lines of 'the errors are binomially distributed'. How can errors be binomially distributed? $\endgroup$ – luciano Mar 26 '14 at 19:13
  • 5
    $\begingroup$ Whatever book you read that in, throw it away. $\endgroup$ – Scortchi - Reinstate Monica Mar 26 '14 at 20:24
7
$\begingroup$

Suppose you observe several nests at different mean daily temperatures $t$. How does the probability $\pi(t)$ of nest success depend on the temperature $t$? (If nests are independent, the number of nests with success at temperature $t$ is then binomially distributed with $n$ equal to the number of nests observed and success probability $\pi(t)$.)

Logistic regression is one approach (using the logistic function) of specifying the success probability as a function of temperature via stretching and shifting the logistic curve, with the amount of stretching and shifting required to be estimated from the data.

$\endgroup$
15
$\begingroup$

Without equations? Yikes. Let's see:

The logistic regression model is literally a model for the $p$ parameter of a binomial distribution; with a continuous predictor, each point can have its own distribution. (In the cases where the observations are 0-1, we deal with the Bernoulli special case; this is a common situation.)

The $n$ is given, not modelled. So the result is, with a model relating the $p_i$'s and a known $n_i$, we can model binomial data in terms of a predictor that describes the mean (and variance) via its model for $p$. The model may be fit via maximum likelihood estimation, but because of its special form (exponential family), ML is relatively "nice".

Because the logistic link is canonical for the binomial family, it's even nicer, since the sufficient statistics are of very simple form - this makes it convenient for dealing with large samples, or even to develop 'online' algorithms.

Of course, $p$, being a probability, lies between 0 and 1. This, naturally, means that when we write a model for it in terms of some other variable, that model should not crash through those limits, so as the independent variable gets sufficiently large or small, the relationship must bend to stay inside the bounds.

With logistic regression, that curve (the link function) is a logistic function. Other functions are possible, and many packages implement several (R has three suitable ones built into its glm functionality if I recall right).


No equality symbols were harmed in the making of this post.

$\endgroup$
2
$\begingroup$

Your model assumes the success of a nest can be viewed as a gamble: God flips a loaded coin with sides labeled "success" and "failure." The outcome of the flip for one nest is independent of the outcome of the flip for any other nest.

The birds do have something going for them, though: the coin might heavily favor success at some temperatures compared to others. Thus, when you have the chance to observe nests at a given temperature, the number of successes equals the number of successful flips of the same coin--the one for that temperature. The corresponding Binomial distribution describes the chances of successes. That is, it establishes the probability of zero successes, of one, of two, ... and so on through the number of nests.

One reasonable estimate of the relationship between the temperature and how God loads the coins is given by the proportion of successes observed at that temperature. This is the Maximum Likelihood estimate (MLE).

For example, suppose you observe $7$ nests at a temperature of $10$ degrees and $3$ of those nests are successful. The MLE is $3/7.$ That is, we estimate that God's coin has a $3/7$ chance of showing success. The corresponding Binomial distribution is plotted in the first row of the figure (see below) under the heading "10 Degrees." It represents the chances with the heights of vertical line segments. The red segment corresponds to the observed value of $3$ successes.

The temperatures must vary in your data. As a running example, let's suppose that at temperatures $5,10,15,20$ degrees you observed $0,3,2,3$ successes among $2,7,5,3$ nests. This dataset is plotted by the gray circles in the "Fit" panels of the figure. The height of a circle represents its success rate. The circle areas are proportional to the numbers of nests (thereby emphasizing the data with more nests).

The top row of the figure shows the MLEs at each of the four observed temperatures. The red curve in the "Fit" panel traces out how the coin is loaded, depending on temperature. By construction, this trace passes through each of the data points. (What it does at intermediate temperatures is unknown; I have crudely connected the values to emphasize this point.)

This "saturated" model is not very useful, precisely because it gives us no basis to estimate how God will load the coins at intermediate temperatures. To do that, we need to suppose there is some kind of "trend" curve that relates coin loadings to temperature.

Figure

The bottom row of the figure fits such a trend. The trend is limited in what it can do: when plotted in appropriate ("log odds") coordinates, as shown in the "Logit Response" panels at left, it can only follow a straight line. Any such straight line determines the loading of the coin at all temperatures, as shown by the corresponding curved line in the "Fit" panels. That loading, in turn, determines the Binomial distributions at all temperatures. The bottom row plots those distributions for the temperatures where nests were observed. (The dashed black lines mark the expected values of the distributions, helping to identify them fairly precisely. You don't see those lines in the top row of the figure because they coincide with the red segments.)

Now a tradeoff must be made: the line might pass closely to some of the data points, only to veer far from others. This causes the corresponding Binomial distribution to assign lower probabilities to most of the observed values than before. You can see this clearly at 10 degrees and 15 degrees: the probability of the observed values is not the highest possible probability, nor is it close to the values assigned in the upper row.

Logistic regression slides and wiggles the possible lines around (in the coordinate system used by the "Logit Response" panels), converts their heights into Binomial probabilities (the "Fit" panels), assesses the chances assigned to the observations (the four right panels), and chooses the line that gives the best combination of those chances.

What is "best"? Simply that the combined probability of all the data is as large as possible. In this way no single probability (the red segments) is allowed to be truly tiny, but usually most of the probabilities will not be as high as they were in the saturated model.

Here is one iteration of the logistic regression search where the line was rotated downwards:

Figure 2

First, notice what stayed the same: the gray points in the "Fit" scatterplot are fixed because they represent the data. Likewise, the ranges of values and the horizontal positions of the red segments in the four Binomial plots also are fixed, because they also represent the data. However, this new line loads the coins in a radically different way. In so doing, it changes the four Binomial distributions (the gray segments). For instance, it gives the coin about a 70% success rate at a temperature of $10$ degrees, corresponding to a distribution whose probabilities are highest for 4 to 6 successes. This line actually does a great job of fitting the data for $15$ degrees but a terrible job of fitting the other data. (At 5 and 20 degrees the Binomial probabilities assigned to the data are so tiny you cannot even see the red segments.) Overall, this is a much worse fit than the ones shown in the first figure.


I hope this discussion has helped you develop a mental image of the Binomial probabilities changing as the line is varied, all the while keeping the data the same. The line fit by logistic regression attempts to make those red bars overall as high as possible. Thus, the relationship between logistic regression and the family of Binomial distributions is deep and intimate.


Appendix: R code to produce the figures

#
# Create example data.
#
X <- data.frame(temperature=c(5,10,15,20),
                nests=c(2,7,5,3),
                successes=c(0,3,2,3))
#
# A function to plot a Binomial(n,p) distribution and highlight the value `k0`.
#
plot.binom <- function(n, p, k0, highlight="#f02020", ...) {
  plot(0:n, dbinom(0:n, n, p), type="h", yaxt="n",
       xlab="Trials", ylab="Probability", ...)
  abline(v = p*n, lty=3, lwd=2)
  if(!missing(k0)) lines(rep(k0,2), c(0, dbinom(k0,n,p)), lwd=2, col=highlight)
}
#
# A function to convert from probability to log odds.
#
logit <- function(p) log(p) - log(1-p)
#
# Fit a saturated model, then the intended model.
#
# Ordinarily the formula for the saturated model would be in the form
# `... ~ factor(temperature)`, but the following method makes it possible to  
# plot the predicted values in a visually effective way.
#
fit.0 <- glm(cbind(successes, nests-successes) ~ factor(round(temperature/5)), 
             data=X, family=binomial)
summary(fit.0)

fit <- glm(cbind(successes, nests-successes) ~ temperature, 
           data=X, family=binomial)
summary(fit)
#
# Plot both fits, one per row.
#
lfits <- list(fit.0, fit)
par.old <- par(mfrow=c(length(lfits), nrow(X)+2))
for (fit in lfits) {
  #
  # Construct arrays of plotting points.
  #
  X$p.hat <- predict(fit, type="response")
  Y <- data.frame(temperature = seq(min(X$temperature), max(X$temperature), 
                                    length.out=101))
  Y$p.hat <- predict(fit, type="response", newdata=Y)  # Probability
  Y$lambda.hat <- predict(fit, type="link", newdata=Y) # Log odds
  #
  # Plot the fit in terms of log odds.
  #
  with(Y, plot(temperature, lambda.hat, type="n", 
               yaxt="n", bty="n", main="Logit Response",
               ylab=expression(hat(lambda))))
  if (isTRUE(diff(range(Y$lambda.hat)) < 6)) {
    # Draw gridlines and y-axis labels
    p <- c( .10, .25, .5, .75, .9)
    q <- logit(p)
    suppressWarnings(rug(q, side=2))
    abline(h=q, col="#d0d0d0")
    mtext(signif(p, 2), at=q, side=2, cex=0.6)
  }
  with(Y, lines(temperature, lambda.hat, lwd=2, col="#f02020"))
  #
  # Plot the data and the fit in terms of probability.
  #
  with(X, plot(temperature, successes/nests, ylim=0:1,
               cex=sqrt(nests), pch=21, bg="Gray",
               main="Fit"))
  with(Y, lines(temperature, p.hat, col="#f02020", lwd=2))
  #
  # Plot the Binomial distributions associated with each row of the data.
  #
  apply(X, 1, function(x) plot.binom(x[2], x[4], x[3], bty="n", lwd=2, col="Gray",
                                     main=paste(x[1], "Degrees")))
}
par(mfrow=par.old)
$\endgroup$
  • $\begingroup$ would be there the r code disclosed? many thanks. $\endgroup$ – Maximilian Dec 4 at 9:29
  • 1
    $\begingroup$ @Maximilian I added the code. $\endgroup$ – whuber Dec 4 at 15:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.