Transformation by subtraction (for continuous data arbitrarily limited in the experiment)

Question

Consider a study looking at the relationship of chili-pepper plant height to number of chilies produced.

Data was collected on all plants taller than 10cm. Initial look at the distribution of heights shows that most plants are on the short end, with a few reaching a max height of 150 cm as seen in a histogram. Taking the log(plantheight) normalizes the distribution somewhat. However, looking at log(plantheight - 10) gives a very nice looking normal distribution to the histogram.

• First, is this a valid transformation? Shifting the values to zero, then taking the log?

• Second, if not, what is the best way to deal with data that had been arbitrarily cut off in the experiment (no values below 10)?

Answer 1

First, why are you transforming at all? You have a tag for "regression" so I'm guessing it's regression. In this case, regression does not require that the dependent variable be normal, it requires that the error (which is estimated by the residuals) is $\mathcal{N}(0,1)$.

Second, if you shift values to 0 and then take the log, you will have a problem $log(0)$ does not exist. If you shift them to something close to 0, then that point is arbitrary (e.g. 0.1? 0.01? Why?)

Third, interpreting the results of a regression of log(x - 9.9) is going to be tricky.

Finally, if you decide that you do need to transform, see @corsario 's suggestions.

Answer 2

I would prefer to use formal rules for transforming-modelling the data.

1. Box-Cox transformation: It is a family of power transformations that includes the logarithm as a limiting case.

2. Use of flexible distributions such as the skew-normal or the two-piece normal, or even more flexible distributions such as the sinh-arcsinh normal. You could also model the log-observations using these distributions.

3. Depending on how the data looks like, perhaps you will need to use a truncated (below 10) model. If you can show us the histogram, we could give you ad hoc suggestions.

4. If you have a reasonable sample size (say, more than 30), you could also use nonparametric models (e.g. kernel density estimators). For example

   plot(density(plantheight))
   plot(density(log(plantheight)))
   

Best wishes.