2
$\begingroup$

I have data from a simple experiments where people put (a fixed number of) balls either to the left or to the right of them (each ball is just the same with regards to consequences of putting them to left or right). The observation / dependent variable is the number of balls put to the right (out of the total amount of balls given to them).

First, I fitted a linear regression trying to explain this sum of balls put to the right based on some predictors. Now a linear regression is obviously not the best choice, since the data is discrete instead of continuous and has natural lower and upper bounds.

Then I fitted a binomial regression to the data (or a GLM with logit link function). The estimates are pretty similar (at least when judging the direction), but there is a huge difference in standard errors on the predictors. In the binomial regression they are about 26 times smaller than in the linear regression.

I was wondering why that is. As outlined, the linear regression is not the best model for the data – but is it so much worse? Or do I violate critical assumptions and my data actually is not generated by a binomial distribution and hence the GLM with binomial link function can not be trusted?

$\endgroup$
  • 3
    $\begingroup$ Binomial is the response distribution, not the link function. The link function, by default, will be the logit. See my answer here: Difference between logit and probit models. $\endgroup$ – gung - Reinstate Monica Aug 14 '15 at 15:43
  • 3
    $\begingroup$ More to the point: what is your link function? Unless it's the identity and you are expressing the responses in the same way in both models, then the parameters aren't comparable in the first place, so "26 times smaller" is meaningless. $\endgroup$ – whuber Aug 14 '15 at 17:26
  • $\begingroup$ Without seeing what your data is and exactly what you did, it's going to be hard to say much more than has already been said. $\endgroup$ – Glen_b Aug 15 '15 at 0:21
2
$\begingroup$

$\newcommand{\fractionBallsRight}{\text{fractionBallsRight}} \newcommand{\BallsToRight }{\text{BallsToRight }} \newcommand{\TotalBalls}{\text{TotalBalls}}$ Let us analyse what happens in both cases. First, your data is in a data table (let's call it Data) with columns: $\BallsToRight, \TotalBalls, x_1, x_2, \ldots, x_n$, where the $x$'s are explanatory variables.

Let's add an extra column: $\fractionBallsRight = \frac{\BallsToRight}{\TotalBalls}$.

Linear regression

In the linear regression case you try to estimate $\fractionBallsRight = \sum_i \beta_i x_i + \beta_0 + \epsilon$ where some assumptions must be fullfilled. In R this can be estimated by:

lm(fractionBallsRight ~ x1 + x2 + .. + xn + 1, data=Data)

Logistic regression for grouped data

Probably it is good to first take a look at Interesting Logistic Regression Idea - Problem: Data not currently in 0/1 form. Any solutions?

What is estimated in this case is the equation $\fractionBallsRight = \frac{1}{1+e^{-(\sum_i \beta_i x_i + \beta_0)}}$. Clearly this is a completely different equation than the one with the linear regression, therefore the coefficients and their standard errors will be different.

The 'logistic regression equation' can be 'linearised', after some manipulations you will find that $\log\!\left( \frac{\BallsToRight}{1-\BallsToRight}\right)=\sum_i \beta_i x_i + \beta_0$, which again shows that the equation for logistic regression and the one for linear regression are completely different (the right hand side is linear in both cases, but the dependent variable is completely different), and therefore the estimated coefficients and their standard errors will also be different.

The R-code for the logistic regression for grouped data is

glm(cbind(BallsToRight, TotalBalls-BallsToRight) ~ x1+x2+ ... + xn+1, 
    data=Data, family=binomial)

Note that, as @gung correctly said (+1), the estimation method for the linear model is OLS and for logistic regression it is maximum likelihood.

It depends on the data, but at first glance I would suggest to use the logistic regression variant for the reasons you mentioned (OLS assumptions are violated).

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Thanks! Trying your GLM function in R gives me Error in model.frame.default(formula = c(right, total_balls) ~ predictor, data = data, : variable lengths differ (found for 'predictor') - which makes sense given the c() statement that generates a vector that is double the length of the number of rows in the data-frame, right? Do you need an additional argument in the function for this type of formula-definition? $\endgroup$ – grueb Aug 14 '15 at 16:20
  • $\begingroup$ @grueb: before "~" you should concatentate two vectors of equal length: the number of balls to the right and the second the number of balls to the left, they must have equal length and the same length as the predictor vector. $\endgroup$ – user83346 Aug 14 '15 at 16:25
  • 1
    $\begingroup$ @grueb, notice that on the left hand side of the formula, he does not have c(right, total_balls), but instead has c(right, total_balls-right). $\endgroup$ – gung - Reinstate Monica Aug 14 '15 at 16:25
  • $\begingroup$ @gung: thanks :-) (another +1 on the comment for helping me) $\endgroup$ – user83346 Aug 14 '15 at 16:26
  • 1
    $\begingroup$ @grueb: sorry, I think I am the source of confusion, the help on 'glm' (?glm) says " or as a two-column matrix with the columns giving the numbers of successes and failures", so probably it must be cbind() in stead of c(), I change the text accordingly $\endgroup$ – user83346 Aug 14 '15 at 16:39
6
$\begingroup$

When you use linear regression, you are violating the assumptions of normality and homoscedasticity (of the residuals). Using a linear regression here cannot be trusted.

The standard errors from the logistic regression will be appropriate. In addition, the way the standard errors are calculated differs. Your linear regression almost certainly used ordinary least squares to fit the model. From there, the standard errors are the square roots of the main diagonal of $\ \hat\sigma^2\bf (X'X)^{-1}$. The logistic regression will have been fit through maximum likelihood, and the Wald standard errors are determined from the degree of curvature at the maximum of the likelihood function. If the response distribution was specified as normal, these would be the same, but since the response distribution is binomial there is no reason they should be the same.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I see all that – still I am puzzled by these 26 folds changes in SE ... a binomial approximates a normal distribution with increasing number of trials, right? Its shape is therefore not so different. So it all goes down to the likelihood function and SE calculation and not the assumption about the distribution of the data? Or is it more about the model how the data were generated? Sorry, I am trying to get my head around this conceptually ... $\endgroup$ – grueb Aug 14 '15 at 17:06
  • 2
    $\begingroup$ I don't know, @grueb. Maybe if I had the data, I could try to figure it out. Exactly how this will play out will differ based on the specific features of your data. Nonetheless, the end result is always going to be that your linear model will be invalid & the logistic regression will be valid. $\endgroup$ – gung - Reinstate Monica Aug 14 '15 at 19:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.