2
$\begingroup$

I am regressing actual counts of traffic against predictions using ridge regression (cv.glmnet in R). The data (both predicted and actual) has a roughly exponential distribution, i.e. a few large values (which are important to predict) and many small ones. Residuals in the model are usually proportional to the size of the target variable.

What is the best approach to fit such a model correctly?

Transform both predicted and target data beforehand (cube root, log, Box-Cox)?

Or is there something I can do with the estimating process that negates the need to do this - by treating errors in large values as less bad than errors in small ones?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Doesn't cv.glmnet support Poisson regression directly? Although in my experience it can fail to converge on moderate to large problems, of all the options available it would be the first method to try with count data. $\endgroup$
    – whuber ♦
    Commented Dec 16, 2015 at 14:39
  • $\begingroup$ I was under the impression a Poisson regression (i.e. log link function) would be wrong if the relationship between my Xs and Y is linear? While both are distributed exponentially I still expect a linear relationship. $\endgroup$
    – Sideshow Bob
    Commented Dec 16, 2015 at 17:49

1 Answer 1

Reset to default
3
$\begingroup$

You're missing a great deal of the point of using glmnet. The residuals don't have to be normal (note that it was always the residuals that need normality in ordinary regression and not the data). Your data are counts and will typically follow a distribution that looks exponential that's called a Poisson distribution. You can set the kind of distribution with the family argument in glmnet. See the help on glmnet as well as cv.glmnet. The ... argument that is available in cv.glmnet means you can pass the same arguments that you do to glmnet where required, as in this case.

Once you've passed the correct arguments the software will handle the generation of the model correctly. However, make certain that you additionally read up on understanding regression coefficients for Poisson regression.

Improve this answer
$\endgroup$
11
  • $\begingroup$ "... an exponential distribution that's called a Poisson distribution" is a contradiction in terms. $\endgroup$
    – whuber ♦
    Commented Dec 16, 2015 at 17:57
  • $\begingroup$ Is a poisson regression still right if the relationship between my Xs and Y is linear? $\endgroup$
    – Sideshow Bob
    Commented Dec 16, 2015 at 18:17
  • $\begingroup$ @whuber Ooops, that wasn't my intent... hopefully it's fixed. $\endgroup$
    – John
    Commented Dec 17, 2015 at 0:27
  • $\begingroup$ Sideshow, you can look at fit of the distribution for your case because I don't know what scale you're looking at linearity (you talked about exponential looking properties of both variables). The most critical thing is that using a Poisson model is dealing with the variance that inflates with count. $\endgroup$
    – John
    Commented Dec 17, 2015 at 0:29
  • 1
    $\begingroup$ No, if the assumption is a normal distribution of residuals, such as if you were using a normal gaussian, then they should be. $\endgroup$
    – John
    Commented Dec 18, 2015 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.