Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Trying to understand how log-transforming a count variable changes the regression equation.
The reason I ask is because I see a lot of researchers log-transform their data so they can use OLS.
Say you have a Poisson GLM:
$$ y = exp(\alpha + \beta x) \implies log(y) = \alpha + \beta x $$
where $y$ is a Poisson random variable with zeros.
Suppose you want to log-transform it with a constant $c$. Does that mean your equation becomes
$$ log(y + c) = \alpha + \beta x + c $$
?
As comments and their links note, there's a difference between a Poisson GLM with a log link and ordinary least squares (OLS) linear regression with a log-transformed count outcome. The Poisson GLM models the log of the expected value of the counts, based on the equality of mean and variance for a Poisson-distributed variable. OLS with log-transformed counts models the expected value of the log of the counts and handles residual error as in OLS, but in the log-transformed scale.
When you add a constant to the observed counts to avoid taking the log of 0, you are changing the effective outcome variable. You are not adding a constant offset to the right-hand side of the equation, as your last equation would seem to imply. If you add, say, 0.5 to a count value of 1, then the difference between log(1.5) and log(1) is about 0.4. If you add 0.5 to a count value of 100, then the difference between log(100.5) and log(100) is about 0.01. That's not a constant offset.
This page discusses ways to handle 0 counts in regression. The square-root transformation doesn't require adding anything to the observed counts, and approximately stabilizes the variance for a Poisson-distributed variable. For linear regression I tend to use square-root for counts, reserving log transformations for necessarily positive values whose measurement error is proportional to the mean.
As @Glen_b explains on the page linked above, the different approaches of a Poisson GLM and linear regression have practical implications:
The canonical [log] link is not generally a particularly good transformation for Poisson data... It's a good 'transformation' for the conditional population mean of a Poisson in a number of contexts, but not always of Poisson data.
He goes on to note that adding a constant of about 0.5 to outcome counts can work well if you nevertheless want to log transform. But other than changing the outcome values somewhat so you avoid 0 counts, it's not a transformation of the underlying regression equation in the way you suggest.
You can see this with the simplest linear model, of the mean value. Set up two large populations of Poisson-distributed values with means of 1 or 4, compare the means of the original values against exp(mean(log(y+0.5))) for 1000 samples of 100 cases each.
set.seed(101)
y1 <- rpois(100000,1)
y4 <- rpois(100000,4)
y1c <- y1+0.5 ## for c = 0.5
y4c <- y4+0.5 ## for c = 0.5
mean(y1)
# [1] 0.99968
mean(y1c)
# [1] 1.49968
mean(y4)
# [1] 4.00483
mean(y4c)
# [1] 4.50483
meanExp1 <- NULL
for (i in 1:1000) meanExp1 <- c(meanExp1,exp(mean(log(sample(y1c,100)))))
mean(meanExp1)
# [1] 1.187415
meanExp4 <- NULL
for (i in 1:1000) meanExp4 <- c(meanExp4,exp(mean(log(sample(y4c,100)))))
mean(meanExp4)
# [1] 4.0049
The negative bias from the y + 0.5 values is obvious, and different depending on the initial distribution of y.
Poisson GLM models of the original count values perform as expected.
meanPoisGLM1 <- NULL
for (i in 1:1000) meanPoisGLM1 <-
c(meanPoisGLM1,exp(coef(glm(sample(y1,100)~1,family=poisson))[[1]]))
mean(meanPoisGLM1)
# [1] 0.99892
meanPoisGLM4 <- NULL
for ( i in 1:1000) meanPoisGLM4 <-
c(meanPoisGLM4,exp(coef(glm(sample(y4,100)~1,family=poisson))[[1]]))
mean(meanPoisGLM4)
# [1] 3.99786
