# How should I transform non-negative data including zeros?

If I have highly skewed positive data I often take logs. But what should I do with highly skewed non-negative data that include zeros? I have seen two transformations used:

• log(x+1) which has the neat feature that 0 maps to 0.
• log(x+c) where c is either estimated or set to be some very small positive value.

Are there any other approaches? Are there any good reasons to prefer one approach over the others?

-
I've summarized some of the answers plus some other material at robjhyndman.com/researchtips/transformations –  Rob Hyndman Aug 13 '10 at 4:28
excellent way to transform and promote stat.stackoverflow ! –  robin girard Aug 13 '10 at 11:49
Yes, I agree @robingirard (I just arrived here now because of Rob's blog post)! –  Feral Oink Jul 19 '12 at 2:26
Also see stats.stackexchange.com/questions/39042/… for an application to left-censored data (which can be characterized, up to a shift of location, exactly as in the present question). –  whuber Oct 9 '12 at 22:05
This Q is also relevant to this discussion: How small a quantity should be added to x to avoid taking the log of zero. –  gung Oct 11 '12 at 13:22

It seems to me that the most appropriate choice of transformation is contingent on the model and the context.

The '0' point can arise from several different reasons each of which may have to be treated differently:

• Truncation (as in Robin's example): Use appropriate models (e.g., mixtures, survival models etc)
• Missing data: Impute data / Drop observations if appropriate.
• Natural zero point (e.g., income levels; an unemployed person has zero income): Transform as needed
• Sensitivity of measuring instrument: Perhaps, add a small amount to data?

I am not really offering an answer as I suspect there is no universal, 'correct' transformation when you have zeros.

-
Every answer to my question has provided useful information and I've up-voted them all. But I can only select one answer and Srikant's provides the best overview IMO. –  Rob Hyndman Aug 13 '10 at 2:25

No-one mentioned the inverse hyperbolic sine transformation. So for completeness I'm adding it here.

This is an alternative to the Box-Cox transformations and is defined by $$f(y,\theta) = \text{sinh}^{-1}(\theta y)/\theta = \log[\theta y + (\theta^2y^2+1)^{1/2}]/\theta,$$ where $\theta>0$. For any value of $\theta$, zero maps to zero. There is also a two parameter version allowing a shift, just as with the two-parameter BC transformation. Burbidge, Magee and Robb (1988) discuss the IHS transformation including estimation of $\theta$.

The IHS transformation works with data defined on the whole real line including negative values and zeros. For large values of $y$ it behaves like a log transformation, regardless of the value of $\theta$ (except 0). The limiting case as $\theta\rightarrow0$ gives $f(y,\theta)\rightarrow y$.

It looks to me like the IHS transformation should be a lot better known than it is.

-

A useful approach when the variable is used as an independent factor in regression is to replace it by two variables: one is a binary indicator of whether it is zero and the other is the value of the original variable or a re-expression of it, such as its logarithm. This technique is discussed in Hosmer & Lemeshow's book on logistic regression (and in other places, I'm sure). Truncated probability plots of the positive part of the original variable are useful for identifying an appropriate re-expression.

When the variable is the dependent one in a linear model, censored regression (like Tobit) can be useful, again obviating the need to produce a started logarithm. This technique is common among econometricians.

-
Is modeling data as a zero-inflated Poisson a special case of this approach? –  David Sep 13 '12 at 15:49
@David, although it seems similar, it's not, because the ZIP is a model of the dependent variable, not the independent variable. –  whuber Sep 13 '12 at 16:04

The log transforms with shifts are special cases of the Box-Cox transformations:

$y(\lambda_{1}, \lambda_{2}) = \begin{cases} \frac {(y+\lambda_{2})^{\lambda_1} - 1} {\lambda_{1}} & \mbox{when } \lambda_{1} \neq 0 \\ \log (y + \lambda_{2}) & \mbox{when } \lambda_{1} = 0 \end{cases}$

These are the extended form for negative values, but also applicable to data containing zeros. Box and Cox (1964) presents an algorithm to find appropriate values for the $\lambda$'s using maximum likelihood. This gives you the ultimate transformation.

A reason to prefer Box-Cox transformations is that they're developed to ensure assumptions for the linear model. There's some work done to show that even if your data cannot be transformed to normality, then the estimated $\lambda$ still lead to a symmetric distribution.

I'm not sure how well this addresses your data, since it could be that $\lambda = (0, 1)$ which is just the log transform you mentioned, but it may be worth estimating the requried $\lambda$'s to see if another transformation is appropriate.

In R, the boxcox.fit function in package geoR will compute the parameters for you.

-
hmm, can't get the latex "begin cases" to accept new lines. :-/ –  ars Aug 9 '10 at 16:44
@ars I fixed the eqns to use begin cases. I hope I did not mangle the eqns in the process. –  user28 Aug 9 '10 at 16:55
@Rob: Oh, sorry. Diggle's geoR is the way to go -- but specify lambda2=TRUE in the arguments to boxcox.fit. (Also updated the answer.) –  ars Aug 10 '10 at 2:01
@gd047: here's a nice reference: elevatorlady.ca/doc/refcard/expressions.html –  ars Aug 13 '10 at 17:45
For anyone who reads this wondering what happened to this function, it is now called boxcoxfit. –  chtfn Aug 2 '12 at 6:14

I'm presuming that zero != missing data, as that's an entirely different question.

When thinking about how to handle zero's in multiple linear regression, I tend to consider how many zeros do we actually have?

Only a couple of zeros

If I have a single zero in a reasonably large data set, I tend to:

1. Remove the point, take logs and fit the model
2. Add a small $c$ to the point, take logs and fit the model

Does the model fit change? What about the parameter values. If the model is fairly robust to the removal of the point, I'll go for quick and dirty approach of adding $c$.

You could make this procedure a bit less crude and use the boxcox method with shifts described in ars' answer.

Large number of zeros

If my data set contains a large number of zeros, then this suggests that simple linear regression isn't the best tool for the job. Instead I would use something like mixture modelling (as suggested by Srikant and Robin).

-

If you want something quick and dirty why not use the square root?

-
And frequently the cube root transformation works well, and allows zeros and negatives. I've found cube root to particularly work well when, for example, the measurement is a volume or a count of particles per unit volume. Cube root would convert it to a linear dimension. A more flexible approach is to fit a restricted cubic spline (natural spline) on the cube root or square root, allowing for a little departure from the assumed form. –  Frank Harrell Jun 17 '11 at 13:36
+1. For a little article on cube roots, see stata-journal.com/article.html?article=st0223 (This will be a free .pdf from first quarter 2014.) –  Nick Cox May 28 '13 at 21:08

I assume you have continuous data.

If the data include zeros this means you have a spike on zero which may be due to some particular aspect of your data. It appears for example in wind energy, wind below 2 m/s produce zero power (it is called cut in) and wind over (something around) 25 m/s also produce zero power (for security reason, it is called cut off). While the distribution of produced wind energy seems continuous there is a spike in zero.

My solution: In this case, I suggest to treat the zeros separately by working with a mixture of the spike in zero and the model you planned to use for the part of the distribution that is continuous (wrt Lebesgue).

-

Since the two-parameter fit Box-Cox has been proposed, here's some R to fit input data, run an arbitrary function on it (e.g. time series forecasting), and then return the inverted output:

# Two-parameter Box-Cox function
boxcox.f <- function(x, lambda1, lambda2) {
if(lambda1!=0) {
return(((x + lambda2) ^ lambda1 - 1) / lambda1)
} else {
return(log(x + lambda2))
}
}

# Two-parameter inverse Box-Cox function
boxcox.inv <- function(x, lambda1, lambda2) {
if(lambda1!=0) {
return((lambda1 * x + 1) ^ (1 / lambda1) - lambda2)
} else {
return(exp(x) - lambda2)
}
}

# Function to Box-Cox transform x, apply function g,
# and return inverted Box-Cox output y
boxcox.fit.apply <- function(x, g) {
require(geoR)
require(plyr)

# Fit lambdas
t <- try(lambda.pair <- boxcoxfit(x, lambda2=T)$lambda) # Estimating both lambdas sometimes fails; if so, estimate lambda1 only if(inherits(t, "try-error")) { lambda1 <- boxcoxfit(x)$lambda
lambda2 <- 0
} else {
lambda1 <- lambda.pair[1]
lambda2 <- lambda.pair[2]
}
x.boxcox <- boxcox.f(x, lambda1, lambda2)

# Apply function g to x.boxcox. This should return data similar to x (e.g. ts)
y <- aaply(x.boxcox, 1, g)

return(boxcox.inv(y, lambda1, lambda2))
}

-

Suppose Y is the amount of money each American spends on a new car in a given year (total purchase price). Y will spike at 0; will have no values at all between 0 and about 12,000; and will take other values mostly in the teens, twenties and thirties of thousands. Predictors would be proxies for the level of need and/or interest in making such a purchase. Need or interest could hardly be said to be zero for individuals who made no purchase; on these scales non-purchasers would be much closer to purchasers than Y or even the log of Y would suggest. In a case much like this but in health care, I found that the most accurate predictions, judged by test-set/training-set crossvalidation, were obtained by, in increasing order,

1. Logistic regression on a binary version of Y,
2. OLS on Y,
3. Ordinal regression (PLUM) on Y binned into 5 categories (so as to divide purchasers into 4 equal-size groups),
4. Multinomial logistic regression on Y binned into 5 categories,
5. OLS on the log(10) of Y (I didn't think of trying the cube root), and
6. OLS on Y binned into 5 categories.

Some will recoil at this categorization of a continuous dependent variable. But although it sacrifices some information, categorizing seems to help by restoring an important underlying aspect of the situation -- again, that the "zeroes" are much more similar to the rest than Y would indicate.

-
You could also split it into two models: the probability of buying a car (binary response), and the value of the car given a purchase. This is the standard practice in many fields, eg insurance, credit risk, etc. –  Hong Ooi Feb 1 at 9:06
@HongOoi - can you suggest any readings on when this approach is and isn't applicable? –  rolando2 Feb 1 at 12:11