1
$\begingroup$

I have a dataset that contains:

  • the counts of successes, $Y_i$
  • the length of observation, $\text{length}_i$
  • few predictors, $X_1, X_2, \text{etc}$

Since the counts are observed at different lengths I need a rate model. For the Poisson regression we achieve this by writing:

$\log(\mu_i /\text{length}_i) = \beta_0 + \beta_1 X_1 +\beta_2X_2$

which is then rewritten as:

$\log(\mu_i) = \beta_0 + \beta_1 X_1 +\beta_2X_2 + \log(\text{length}_i)$

The $\text{length}_i$ is the offset and it has a fixed coefficient of 1. In R we specify it using the offset method. Something like this:

model <- glm(Y ~ offset(log(LENGTH)) + x1 + x2, data = data, family = "poisson")

However, I get better prediction results when I specify the offset as having a coefficient -0.5. That is:

model <- glm(Y ~ offset(-0.5 * log(LENGTH)) + x1 + x2, data = data, family = "poisson")

This would mean that the mean is related to the predictor variables like:

$\log(\mu_i * \sqrt{\text{length}_i}) = \beta_0 + \beta_1 X_1 +\beta_2X_2$

I have a few questions:

  1. Why am I getting better results using negative coefficient?
  2. What are the possible theoretical explanation for a negative offset?
  3. Should I try some other model?
$\endgroup$
  • $\begingroup$ I have a question for you. You said that you are getting "better prediction results". So you are comparing these two models. But this means that you are fitting your Poisson-regression model into two different response variables. One is $log(Y/L)$ and the other one is $log(Y\times\sqrt{L})$. In other words, your data-sets are not the same. So I think, it does not make any sense to compare like that. $\endgroup$ – Stat Jan 12 '14 at 2:42
  • $\begingroup$ The prediction results are better no mater the dataset. I tried predicting on 4 different dataset (including the original dataset) and in all cases the results are better. It seems that $log(Y*\sqrt{L})$ is a better response variable but it is difficult to explain why this is the case. $\endgroup$ – user2840286 Jan 12 '14 at 3:05
1
$\begingroup$

In general if you're interested in predicting a count then there's nothing problematic about having log(exposure) as just another covariate and allowing its coefficient to be estimated from the data. Although it's probably clearer to just drop to say

 model <- glm(Y ~ log(LENGTH) + x1 + x2, data = data, family = "poisson")

A simple case where this would be the right thing to do is when the underlying rate is not constant over exposure.

Also, you don't say what your measure of predictive success is. I'm wondering if the counts that you are predicting are relatively low and your success measure is something symmetrical, like MSE. In that situation, prediction may be better according to the criterion if the model generates forecasts that are actually biased.

I'd guess the things to do is to check that the the curious 0.5 business is still preferred if the criterion is probability of the data, or some non-symmetrical measure.

Finally (and assuming when you say predict you are doing out-of-sample prediction) it's possible that you are overfitting on the training data and would predict better if you had managed not to.

$\endgroup$
  • $\begingroup$ I did try adding the log(LENGTH) but the estimated coefficient is about +0.36. The data is very messy and large dataset (30,000 points). Both the counts and the length are very skewed distributions. The counts has a very large amount of excess of zeroes: ` Min. 1st Qu. Median Mean 3rd Qu. Max. ` ` 0.0000 0.0000 0.0000 0.5059 1.0000 131.0000` $\endgroup$ – user2840286 Jan 13 '14 at 0:12
  • $\begingroup$ Also, I am doing out of sample prediction that is I fit the model on one dataset and use it to predict on three other datasets. I also checked the predictions on the same dataset on which the model was fit and the results are consistently better with a negative coefficient. Can you tell me more about forecasts that are biased. What is it and how can I incorporate it. Unfortunately, I can't disclose the dataset but I can give an example that might help. Let's say the LENGTH is number of words in a file and the count is number of curse words. $\endgroup$ – user2840286 Jan 13 '14 at 0:19
  • $\begingroup$ My measure of better prediction would be to rank the files so that we can find the highest number of obscene words by examining as little words as possible. In other words, I am more interested in files with high density of obscene words. $\endgroup$ – user2840286 Jan 13 '14 at 0:21
  • $\begingroup$ You haven't yet said what doing better means. You fit the model to one set of data, then you apply the model to new covariates, get the expected count and compare it to the observed count... how? $\endgroup$ – conjugateprior Jan 13 '14 at 23:42
1
$\begingroup$

Unfortunately, you've gotten the model itself wrong. With general linear models, you're modeling the mean demand, not the variate itself; the link between the two is the assumed probability distribution of the variate given the mean demand. Hence, defining $\mu_i = \mathbb{E}Y_i$, the model would be:

$\log \mu_i - \log \text{length}_i = \beta_0 + \beta_1X_1 + \beta_2X_2$

If you run the glm with length as a right-hand-side variable, instead of an offset, you can test to see if the coefficient is significantly different than -1. If it isn't, then you don't have a problem. If it is, then you may have just gotten unlucky with your data set, or your model may be misspecified in such a way that the coefficient contains information, or there's overdispersion, in which case the standard errors of your parameter estimates are biased. You can usually capture some, if not all, of this effect by running a negative binomial regression instead. That won't change the parameter estimates, but it will change their standard errors, perhaps enough so that the coefficient of length is no longer significantly different than -1.

If the coefficient remains significantly different than -1, you'll have to think carefully about what is actually being modeled, and how the data is being recorded. There are too many possible causes to list without that knowledge (which I don't have.)

$\endgroup$
  • $\begingroup$ You are right about the model. I did more reading and corrected the equations. I also tried using the negative binomial regression and the results are the same. I always get better prediction when I set the coefficient for $length$ to be -0.5. If I allow the model to estimate it estimates it to be 0.36. $\endgroup$ – user2840286 Jan 12 '14 at 22:44
  • $\begingroup$ What are you modeling? It might help my speculation as to why this would be if I knew that... $\endgroup$ – jbowman Jan 13 '14 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.