# How should I transform non-negative data including zeros?

If I have highly skewed positive data I often take logs. But what should I do with highly skewed non-negative data that include zeros? I have seen two transformations used:

• $\log(x+1)$ which has the neat feature that 0 maps to 0.
• $\log(x+c)$ where c is either estimated or set to be some very small positive value.

Are there any other approaches? Are there any good reasons to prefer one approach over the others?

• I've summarized some of the answers plus some other material at robjhyndman.com/researchtips/transformations – Rob Hyndman Aug 13 '10 at 4:28
• excellent way to transform and promote stat.stackoverflow ! – robin girard Aug 13 '10 at 11:49
• Yes, I agree @robingirard (I just arrived here now because of Rob's blog post)! – Ellie Kesselman Jul 19 '12 at 2:26
• Also see stats.stackexchange.com/questions/39042/… for an application to left-censored data (which can be characterized, up to a shift of location, exactly as in the present question). – whuber Oct 9 '12 at 22:05
• It seems strange to ask about how to transform without having stated the purpose of transforming in the first place. What is the situation? Why is it necessary to transform? If we don't know what you're trying to achieve, how can one reasonably suggest anything? (Clearly one cannot hope to transform to normality, because the existence of a (non-zero) probability of exact zeros implies a spike in the distribution at zero, which spike no transformation will remove -- it can only move it around.) – Glen_b Jul 6 '16 at 6:00

It seems to me that the most appropriate choice of transformation is contingent on the model and the context.

The '0' point can arise from several different reasons each of which may have to be treated differently:

• Truncation (as in Robin's example): Use appropriate models (e.g., mixtures, survival models etc)
• Missing data: Impute data / Drop observations if appropriate.
• Natural zero point (e.g., income levels; an unemployed person has zero income): Transform as needed
• Sensitivity of measuring instrument: Perhaps, add a small amount to data?

I am not really offering an answer as I suspect there is no universal, 'correct' transformation when you have zeros.

• Every answer to my question has provided useful information and I've up-voted them all. But I can only select one answer and Srikant's provides the best overview IMO. – Rob Hyndman Aug 13 '10 at 2:25
• Also note that there are zero-inflated models (extra zeroes and you care about some zeroes: a mixture model), and hurdle models (zeroes and you care about non-zeroes: a two-stage model with an initial censored model). – Wayne Jul 5 '16 at 15:45

No-one mentioned the inverse hyperbolic sine transformation. So for completeness I'm adding it here.

This is an alternative to the Box-Cox transformations and is defined by $$f(y,\theta) = \text{sinh}^{-1}(\theta y)/\theta = \log[\theta y + (\theta^2y^2+1)^{1/2}]/\theta,$$ where $\theta>0$. For any value of $\theta$, zero maps to zero. There is also a two parameter version allowing a shift, just as with the two-parameter BC transformation. Burbidge, Magee and Robb (1988) discuss the IHS transformation including estimation of $\theta$.

The IHS transformation works with data defined on the whole real line including negative values and zeros. For large values of $y$ it behaves like a log transformation, regardless of the value of $\theta$ (except 0). The limiting case as $\theta\rightarrow0$ gives $f(y,\theta)\rightarrow y$.

It looks to me like the IHS transformation should be a lot better known than it is.

• Looks like a good alternative to $tanh$/logistic transformations – Firebug Jul 5 '16 at 12:18
• About the IHS some seems to disagree: onlinelibrary.wiley.com/doi/10.1890/10-0340.1/abstract – kjetil b halvorsen Dec 26 '16 at 2:04
• That paper is about the inverse sine transformation, not the inverse hyperbolic sine. – Bryan Dec 30 '18 at 13:14

A useful approach when the variable is used as an independent factor in regression is to replace it by two variables: one is a binary indicator of whether it is zero and the other is the value of the original variable or a re-expression of it, such as its logarithm. This technique is discussed in Hosmer & Lemeshow's book on logistic regression (and in other places, I'm sure). Truncated probability plots of the positive part of the original variable are useful for identifying an appropriate re-expression. (See the analysis at https://stats.stackexchange.com/a/30749/919 for examples.)

When the variable is the dependent one in a linear model, censored regression (like Tobit) can be useful, again obviating the need to produce a started logarithm. This technique is common among econometricians.

• Is modeling data as a zero-inflated Poisson a special case of this approach? – David LeBauer Sep 13 '12 at 15:49
• @David, although it seems similar, it's not, because the ZIP is a model of the dependent variable, not the independent variable. – whuber Sep 13 '12 at 16:04
• @whuber This technique is discussed in Hosmer & Lemeshow's book on logistic regression By any chance would you happen to know in which chapter they discuss this technique? I'm looking at their book, but can't seem to find the right page... – landroni Jul 6 '15 at 18:08
• @landroni H&L was fresh in my mind back then, so I feel confident there's something in that book related to this topic. (I had consulted with it to develop some very large regression models and many of the independent variables had to be treated in this way.) However, I can't find the reference at this point, either. I have written about this technique in subsequent posts, in case you're looking for details. Two that show up in a site search are at stats.stackexchange.com/questions/6563 and stats.stackexchange.com/questions/4831. – whuber Jul 6 '15 at 18:32
• @landroni Yes, they are equivalent, in the same way that all numerical encodings of any binary variable are equivalent. Choose whichever one you find most convenient to interpret. – whuber Jul 6 '15 at 21:24

The log transforms with shifts are special cases of the Box-Cox transformations:

$y(\lambda_{1}, \lambda_{2}) = \begin{cases} \frac {(y+\lambda_{2})^{\lambda_1} - 1} {\lambda_{1}} & \mbox{when } \lambda_{1} \neq 0 \\ \log (y + \lambda_{2}) & \mbox{when } \lambda_{1} = 0 \end{cases}$

These are the extended form for negative values, but also applicable to data containing zeros. Box and Cox (1964) presents an algorithm to find appropriate values for the $\lambda$'s using maximum likelihood. This gives you the ultimate transformation.

A reason to prefer Box-Cox transformations is that they're developed to ensure assumptions for the linear model. There's some work done to show that even if your data cannot be transformed to normality, then the estimated $\lambda$ still lead to a symmetric distribution.

I'm not sure how well this addresses your data, since it could be that $\lambda = (0, 1)$ which is just the log transform you mentioned, but it may be worth estimating the requried $\lambda$'s to see if another transformation is appropriate.

In R, the boxcox.fit function in package geoR will compute the parameters for you.

• @ars. I think the boxcox function in MASS only estimates $\lambda_1$ and assumes $\lambda_2=0$. – Rob Hyndman Aug 10 '10 at 1:27
• @Rob: Oh, sorry. Diggle's geoR is the way to go -- but specify lambda2=TRUE in the arguments to boxcox.fit. (Also updated the answer.) – ars Aug 10 '10 at 2:01
• For anyone who reads this wondering what happened to this function, it is now called boxcoxfit. – stragu Aug 2 '12 at 6:14

I'm presuming that zero != missing data, as that's an entirely different question.

When thinking about how to handle zeros in multiple linear regression, I tend to consider how many zeros do we actually have?

Only a couple of zeros

If I have a single zero in a reasonably large data set, I tend to:

1. Remove the point, take logs and fit the model
2. Add a small $$c$$ to the point, take logs and fit the model

Does the model fit change? What about the parameter values? If the model is fairly robust to the removal of the point, I'll go for quick and dirty approach of adding $$c$$.

You could make this procedure a bit less crude and use the boxcox method with shifts described in ars' answer.

Large number of zeros

If my data set contains a large number of zeros, then this suggests that simple linear regression isn't the best tool for the job. Instead I would use something like mixture modelling (as suggested by Srikant and Robin).

If you want something quick and dirty why not use the square root?

• And frequently the cube root transformation works well, and allows zeros and negatives. I've found cube root to particularly work well when, for example, the measurement is a volume or a count of particles per unit volume. Cube root would convert it to a linear dimension. A more flexible approach is to fit a restricted cubic spline (natural spline) on the cube root or square root, allowing for a little departure from the assumed form. – Frank Harrell Jun 17 '11 at 13:36
• +1. For a little article on cube roots, see stata-journal.com/article.html?article=st0223 (This will be a free .pdf from first quarter 2014.) – Nick Cox May 28 '13 at 21:08
• A square root of zero, is zero, so only the non-zeroes values are transformed. This does nothing to deal with the spike, if zero inflated, and can cause serious problems if, in groups, each has a different amount of zeroes. In other words, if some groups have many zeroes and others have few, this transformation can affect many things in a negative way. For the group with the largest variance (also had the least zeroes), almost all values are being transformed. In contrast, those with the most zeroes, not much of the values are transformed. This can change which group has the largest variance. – D_Williams Dec 26 '16 at 16:08
• No transformation will maintain the variance in the case described by @D_Williams. Mixture models (mentioned elsewhere in this thread) would probably be a good approach in that case. – mkt - Reinstate Monica Sep 19 '19 at 16:36

I assume you have continuous data.

If the data include zeros this means you have a spike on zero which may be due to some particular aspect of your data. It appears for example in wind energy, wind below 2 m/s produce zero power (it is called cut in) and wind over (something around) 25 m/s also produce zero power (for security reason, it is called cut off). While the distribution of produced wind energy seems continuous there is a spike in zero.

My solution: In this case, I suggest to treat the zeros separately by working with a mixture of the spike in zero and the model you planned to use for the part of the distribution that is continuous (wrt Lebesgue).

Comparing the answer provided in by @RobHyndman to a log-plus-one transformation extended to negative values with the form:

$$T(x) = \text{sign}(x) \cdot \log{\left(|x|+1\right)}$$

r = -1000:1000

l = sign(r)*log1p(abs(r))
l = l/max(l)
plot(r, l, type = "l", xlab = "Original", ylab = "Transformed", col = adjustcolor("red", alpha = 0.5), lwd = 3)

#We scale both to fit (-1,1)
for(i in exp(seq(-10, 100, 10))){
s = asinh(i*r)

s = s / max(s)
lines(r, s, col = adjustcolor("blue", alpha = 0.2), lwd = 3)
}
legend("topleft", c("asinh(x)", "sign(x) log(abs(x)+1)"), col = c("blue", "red"), lty = 1)


As you can see, as $\theta$ increases more the transform looks like a step function. With $\theta \approx 1$ it looks a lot like the log-plus-one transformation. And when $\theta \rightarrow 0$ it approaches a line.

EDIT: Keep in mind the log transform can be similarly altered to arbitrary scale, with similar results. I just wanted to show what $\theta$ gives similar results based on the previous answer. The biggest difference between both approaches is the region near $x=0$, as we can see by their derivatives.

Since the two-parameter fit Box-Cox has been proposed, here's some R to fit input data, run an arbitrary function on it (e.g. time series forecasting), and then return the inverted output:

# Two-parameter Box-Cox function
boxcox.f <- function(x, lambda1, lambda2) {
if (lambda1!=0) {
return(((x + lambda2) ^ lambda1 - 1) / lambda1)
} else {
return(log(x + lambda2))
}
}

# Two-parameter inverse Box-Cox function
boxcox.inv <- function(x, lambda1, lambda2) {
if (lambda1!=0) {
return((lambda1 * x + 1) ^ (1 / lambda1) - lambda2)
} else {
return(exp(x) - lambda2)
}
}

# Function to Box-Cox transform x, apply function g,
# and return inverted Box-Cox output y
boxcox.fit.apply <- function(x, g) {
require(geoR)
require(plyr)

# Fit lambdas
t <- try(lambda.pair <- boxcoxfit(x, lambda2=T)$lambda) # Estimating both lambdas sometimes fails; if so, estimate lambda1 only if (inherits(t, "try-error")) { lambda1 <- boxcoxfit(x)$lambda
lambda2 <- 0
} else {
lambda1 <- lambda.pair[1]
lambda2 <- lambda.pair[2]
}
x.boxcox <- boxcox.f(x, lambda1, lambda2)

# Apply function g to x.boxcox. This should return data similar to x (e.g. ts)
y <- aaply(x.boxcox, 1, g)

return(boxcox.inv(y, lambda1, lambda2))
}


The Yeo-Johnson power transformation discussed here has excellent properties designed to handle zeros and negatives while building on the strengths of Box Cox power transformation. This is what I typically go to when I am dealing with zeros or negative data.

Here is a summary of transformations with pros/cons to illustrate why Yeo-Johnson is preferable.

Log

Pros: Does well with positive data.

Cons: Does not handle zeros.

> log(0)
[1] -Inf


Log Plus 1

Pros: The plus 1 offset adds the ability to handle zeros in addition to positive data.

Cons: Fails with negative data

> log1p(-1)
[1] -Inf
> log1p(-2)
[1] NaN
Warning message:
In log1p(-2) : NaNs produced


Square Root

Pros: Uses a power transformation that can handle zeros and positive data.

Cons: Fails with negative data

> sqrt(-1)
[1] NaN
Warning message:
In sqrt(-1) : NaNs produced


Box Cox

R Code:

box_cox <- function(x, lambda) {

eps <- 0.00001
if (abs(lambda) < eps)
log(x)
else
(x ^ lambda - 1) / lambda

}


Pros: Enables scaled power transformations

Cons: Suffers from issues with zeros and negatives (i.e. can only handle positive data.

> box_cox(0, lambda = 0)
[1] -Inf
> box_cox(0, lambda = -0.5)
[1] -Inf
> box_cox(-1, lambda = 0.5)
[1] NaN


Yeo Johnson

R Code:

yeo_johnson <- function(x, lambda) {

eps <- .000001
not_neg <- which(x >= 0)
is_neg  <- which(x < 0)

not_neg_trans <- function(x, lambda) {
if (abs(lambda) < eps) log(x + 1)
else ((x + 1) ^ lambda - 1) / lambda
}

neg_trans <- function(x, lambda) {
if (abs(lambda - 2) < eps) - log(-x + 1)
else - ((-x + 1) ^ (2 - lambda) - 1) / (2 - lambda)
}

x[not_neg] <- not_neg_trans(x[not_neg], lambda)

x[is_neg] <- neg_trans(x[is_neg], lambda)

return(x)

}


Pros: Can handle positive, zero, and negative data.

Cons: None that I can think of. Properties are very similar to Box-Cox but can handle zero and negative data.

> yeo_johnson(0, lambda = 0)
[1] 0
> yeo_johnson(0, lambda = -0.5)
[1] 0
> yeo_johnson(-1, lambda = 0.5)
[1] -1.218951

• Cons for Yeo–Johnson: complex, separate transformation for positives and negatives and for values on either side of lambda, magical tuning value (epsilon; and what is lambda?). No readily apparent advantage compared to the simpler negative-extended log transformation shown in Firebug’s answer, unless you require scaled power transformations (as in Box–Cox). – Konrad Rudolph Sep 25 '18 at 11:19
• Cons for Log(x+1): it is arbitrary and rarely is the best choice. See stats.stackexchange.com/questions/30728. – whuber Feb 25 at 21:11

Suppose Y is the amount of money each American spends on a new car in a given year (total purchase price). Y will spike at 0; will have no values at all between 0 and about 12,000; and will take other values mostly in the teens, twenties and thirties of thousands. Predictors would be proxies for the level of need and/or interest in making such a purchase. Need or interest could hardly be said to be zero for individuals who made no purchase; on these scales non-purchasers would be much closer to purchasers than Y or even the log of Y would suggest. In a case much like this but in health care, I found that the most accurate predictions, judged by test-set/training-set crossvalidation, were obtained by, in increasing order,

1. Logistic regression on a binary version of Y,
2. OLS on Y,
3. Ordinal regression (PLUM) on Y binned into 5 categories (so as to divide purchasers into 4 equal-size groups),
4. Multinomial logistic regression on Y binned into 5 categories,
5. OLS on the log(10) of Y (I didn't think of trying the cube root), and
6. OLS on Y binned into 5 categories.

Some will recoil at this categorization of a continuous dependent variable. But although it sacrifices some information, categorizing seems to help by restoring an important underlying aspect of the situation -- again, that the "zeroes" are much more similar to the rest than Y would indicate.

• You could also split it into two models: the probability of buying a car (binary response), and the value of the car given a purchase. This is the standard practice in many fields, eg insurance, credit risk, etc. – Hong Ooi Feb 1 '14 at 9:06
• @HongOoi - can you suggest any readings on when this approach is and isn't applicable? – rolando2 Feb 1 '14 at 12:11

To clarify how to deal with the log of zero in regression models, we have written a pedagogical paper explaining the best solution and the common mistakes people make in practice. We also came out with a new solution to tackle this issue.

You can find the paper by clicking here: https://ssrn.com/abstract=3444996

First, we think that ones should wonder why using a log transformation. In regression models, a log-log relationship leads to the identification of an elasticity. Indeed, if $$\log(y) = \beta \log(x) + \varepsilon$$, then $$\beta$$ corresponds to the elasticity of $$y$$ to $$x$$. The log can also linearize a theoretical model. It can also be used to reduce heteroskedasticity. However, in practice, it often occurs that the variable taken in log contains non-positive values.

A solution that is often proposed consists in adding a positive constant c to all observations $$Y$$ so that $$Y + c > 0$$. However, contrary to linear regressions, log-linear regressions are not robust to linear transformation of the dependent variable. This is due to the non-linear nature of the log function. Log transformation expands low values and squeezes high values. Therefore, adding a constant will distort the (linear) relationship between zeros and other observations in the data. The magnitude of the bias generated by the constant actually depends on the range of observations in the data. For that reason, adding the smallest possible constant is not necessarily the best worst solution.

In our article, we actually provide an example where adding very small constants is actually providing the highest bias. We provide derive an expression of the bias.

Actually, Poisson Pseudo Maximum Likelihood (PPML) can be considered as a good solution to this issue. One has to consider the following process:

$$y_i = a_i \exp(\alpha + x_i' \beta)$$ with $$E(a_i | x_i) = 1$$

This process is motivated by several features. First, it provides the same interpretation to $$\beta$$ as a semi-log model. Second, this data generating process provides a logical rationalization of zero values in the dependent variable. This situation can arise when the multiplicative error term, $$a_i$$ , is equal to zero. Third, estimating this model with PPML does not encounter the computational difficulty when $$y_i = 0$$. Under the assumption that $$E(a_i|x_i) = 1$$, we have $$E( y_i - \exp(\alpha + x_i' \beta) | x_i) = 0$$. We want to minimize the quadratic error of this moment, leading to the following first-order conditions:

$$\sum_{i=1}^N ( y_i - \exp(\alpha + x_i' \beta) )x_i' = 0$$

These conditions are defined even when $$y_i = 0$$. These first-order conditions are numerically equivalent to those of a Poisson model, so it can be estimated with any standard statistical software.

Finally, we propose a new solution that is also easy to implement and that provides unbiased estimator of $$\beta$$. One simply need to estimate:

$$\log( y_i + \exp (\alpha + x_i' \beta)) = x_i' \beta + \eta_i$$

We show that this estimator is unbiased and that it can simply be estimated with GMM with any standard statistical software. For instance, it can be estimated by executing just one line of code with Stata.

We hope that this article can help and we'd love to get feedback from you.

Christophe Bellégo and Louis-Daniel Pape CREST - Ecole Polytechnique - ENSAE

I had the same problem with data and no transformation would give reasonable distribution. I came up with the following idea. I would appreciate if someone decide whether it is worth utilising as I am not a statistitian.

We may adopt the assumption that 0 is not equal to 0. There is a hidden continuous value which we observe as zeros but, the low sensitivity of the test gives any values more than 0 only after reaching the treshold. So what we observe is more like half-normal distribution where all the left side of normal distribution is shown as one rectangle (x=0) in histogram. So maybe we can just perform following steps:

1. Transform the variable to dychotomic values (0 are still zeros, and >0 we code as 1)
2. We search for another continuous variable with high Spearman correlation coefficent with our original variable.
3. We perform logistic regression which predicts 1. Dependant variable - dychotomic, independant - highly correlated variable
4. We look at predicted values for observed zeros in logistic regression.
5. We recode zeros in original variable for predicted in logistic regression. We leave original values higher than 0 intact (however they must be higher than 1)
6. We rank the original variable with recoded zeros.
7. We normalize the ranked variable with Blom - f(r) = vnormal((r+3/8)/(n+1/4); 0;1) where r is a rank; n - number of cases, or Tukey transformation.