4
$\begingroup$

I have created a model to predict the number of people with a certain characteristic (Y) based on predictor variables $X_1$, $X_2$, $X_3$, $X_4$. The model is a multiple linear regression and both the predictors and the outcome variable have been log transformed, that is my equation looks like: $ln(Y)=a +b*ln(X1)+c*ln(X2)+...$

The aim of the model is to then be applied to a dataset for which we have $X_1,X_2,X_3,X_4$ but need to predict Y (in it's original form). Therefore, I need to backtransform the outputs for Y from the model. I understand that a simple reversal of $e^{ln{Y}}$ isn't appropriate as this does not take into account the error terms within the model, and so including a correction for this is necessary.

I have tried a correction term of the form $exp{0.5*variance}$ as per Miller's bias correction using the below code but this gives me wildly unlikely outputs and so I have assumed is not correct:

exp((summary(model)$sigma)**2)*exp(data$model.prediction)

where

  • data is the dataset used
  • model.prediction is the outcome for each case based on the model,
  • The model is called model.

I am struggling to find the correct R code to make this correction. Is there a package that supports back transformation and if so does anyone know what this code is and how to compute the input terms within it?

$\endgroup$
  • 2
    $\begingroup$ In your code, you forgot to half the residual variance. $\endgroup$ – COOLSerdash Aug 10 '18 at 12:37
  • 1
    $\begingroup$ Thanks for spotting this - I think the power two threw me off! I have added the 0.5 multiplier into that exponential and it has definitely helped to get much more accurate results! $\endgroup$ – Hannah Aug 10 '18 at 13:49
  • $\begingroup$ @COOLSerdash Can't believe I missed that. $\endgroup$ – The Laconic Aug 10 '18 at 16:41
4
$\begingroup$

You didn't give any details about why you think the outputs are wildly unlikely, but my guess is that your errors are not normally distributed. That "smearing adjustment" (bias correction) you're using is only valid if the errors are normal.

There is a more general smearing adjustment you can use, which is easy to implement. If I recall correctly, and I think I do, the steps are:

  1. Compute $\exp(X\hat{\beta})$, i.e. the retransformed but unadjusted prediction.
  2. Regress $Y$ against $\exp(X\hat{\beta})$ without an intercept. Call the resulting regression coefficient $\gamma$.
  3. Compute the adjusted retransformed prediction as $\gamma \exp(X\hat{\beta})$.

It's about the most intuitive thing you can do--forget the theory based on the normal distribution and just estimate the multiplier that gets the job done.

For details, see Duan, Naihua. “Smearing Estimate: A Nonparametric Retransformation Method.” Journal of the American Statistical Association, vol. 78, no. 383, 1983, pp. 605–610. JSTOR, JSTOR, www.jstor.org/stable/2288126.

$\endgroup$
  • $\begingroup$ Hi The Laconic - thanks so much for this response. I had indeed assumed normality of residuals. If I were to check the normality I assume the simplest way of confirming this would be to plot the residuals and assess the shape of their distribution. Is there any way to assess the normality in R using something more rigorous i.e some test I can make? Assuming it's not normal it looks from the above like I essential want to assess the magnitude of the disparity between the observed Y and the back-transformed prediction and then apply that to the new data? I'll take a read of the reference now! $\endgroup$ – Hannah Aug 10 '18 at 13:51
  • $\begingroup$ You can look at a histogram of the residuals along with the density of a normal distribution with the same mean and variance; you can look at a quantile-quantile plot; you can run a Jarque Bera test (among others I am sure). $\endgroup$ – The Laconic Aug 10 '18 at 16:44
  • $\begingroup$ Great - thanks so much for confirming! This has been so helpful :) $\endgroup$ – Hannah Aug 14 '18 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.