Negative Offset in Rate (Poisson or Negative Binomial) models

Question

I have a dataset that contains:

• the counts of successes, $Y_i$
• the length of observation, $\text{length}_i$
• few predictors, $X_1, X_2, \text{etc}$

Since the counts are observed at different lengths I need a rate model. For the Poisson regression we achieve this by writing:

$\log(\mu_i /\text{length}_i) = \beta_0 + \beta_1 X_1 +\beta_2X_2$

which is then rewritten as:

$\log(\mu_i) = \beta_0 + \beta_1 X_1 +\beta_2X_2 + \log(\text{length}_i)$

The $\text{length}_i$ is the offset and it has a fixed coefficient of 1. In R we specify it using the offset method. Something like this:

model <- glm(Y ~ offset(log(LENGTH)) + x1 + x2, data = data, family = "poisson")

However, I get better prediction results when I specify the offset as having a coefficient -0.5. That is:

model <- glm(Y ~ offset(-0.5 * log(LENGTH)) + x1 + x2, data = data, family = "poisson")

This would mean that the mean is related to the predictor variables like:

$\log(\mu_i * \sqrt{\text{length}_i}) = \beta_0 + \beta_1 X_1 +\beta_2X_2$

I have a few questions:

1. Why am I getting better results using negative coefficient?
2. What are the possible theoretical explanation for a negative offset?
3. Should I try some other model?

Answer 1

In general if you're interested in predicting a count then there's nothing problematic about having log(exposure) as just another covariate and allowing its coefficient to be estimated from the data. Although it's probably clearer to just drop to say

 model <- glm(Y ~ log(LENGTH) + x1 + x2, data = data, family = "poisson")

A simple case where this would be the right thing to do is when the underlying rate is not constant over exposure.

Also, you don't say what your measure of predictive success is. I'm wondering if the counts that you are predicting are relatively low and your success measure is something symmetrical, like MSE. In that situation, prediction may be better according to the criterion if the model generates forecasts that are actually biased.

I'd guess the things to do is to check that the the curious 0.5 business is still preferred if the criterion is probability of the data, or some non-symmetrical measure.

Finally (and assuming when you say predict you are doing out-of-sample prediction) it's possible that you are overfitting on the training data and would predict better if you had managed not to.

Answer 2

Unfortunately, you've gotten the model itself wrong. With general linear models, you're modeling the mean demand, not the variate itself; the link between the two is the assumed probability distribution of the variate given the mean demand. Hence, defining $\mu_i = \mathbb{E}Y_i$, the model would be:

$\log \mu_i - \log \text{length}_i = \beta_0 + \beta_1X_1 + \beta_2X_2$

If you run the glm with length as a right-hand-side variable, instead of an offset, you can test to see if the coefficient is significantly different than -1. If it isn't, then you don't have a problem. If it is, then you may have just gotten unlucky with your data set, or your model may be misspecified in such a way that the coefficient contains information, or there's overdispersion, in which case the standard errors of your parameter estimates are biased. You can usually capture some, if not all, of this effect by running a negative binomial regression instead. That won't change the parameter estimates, but it will change their standard errors, perhaps enough so that the coefficient of length is no longer significantly different than -1.

If the coefficient remains significantly different than -1, you'll have to think carefully about what is actually being modeled, and how the data is being recorded. There are too many possible causes to list without that knowledge (which I don't have.)