2
$\begingroup$

enter image description here

I basically don't know how to proceed with my transformed dataset.

I have a dataset of 500 observations, positive skew. I have used the bestNormalize package to normalise it (it chose Box-Cox, if that matters) and so now I have a nice dataset of new values, normal distribution. (The result is in a bestNormalise object, but I have also made a df from it, which is what you can see below.)

How do I calculate probabilities for new "original values" from this? I.e 113883785 equals 0.17045703 in the transformed data, and I can use pnorm to calculate the probability for 0.17045703, which should be the probability for 113883785 in the original dataset. But how do I go about getting the probability for 130000, or 586, or any number not exactly in the original dataset? Help would be much appreciated. Thank you!

$\endgroup$
8
  • $\begingroup$ What Box-Cox transformation did you choose? (Box-Cox isn't just one transformation.) $\endgroup$
    – Dave
    Commented Sep 13, 2021 at 20:05
  • $\begingroup$ Thank you for responding @Dave. I did not choose - the package that I have used (bestNormalize) has a function where it tries a number of transformations and picks the best one to normalize the data. The documentation says "Perform a Box-Cox transformation and center/scale a vector to attempt normalization". (cran.r-project.org/web/packages/bestNormalize/bestNormalize.pdf) $\endgroup$
    – Reader 123
    Commented Sep 13, 2021 at 20:34
  • $\begingroup$ The Box-Cox transformation has a parameter. What is the value of that parameter? $\endgroup$
    – Dave
    Commented Sep 13, 2021 at 20:35
  • $\begingroup$ @Dave, is it the lambda from: "Standardized Box Cox Transformation with 500 nonmissing obs.: Estimated statistics: - lambda = 0.2468158 - mean (before standardization) = 373.0962 - sd (before standardization) = 102.0227 " ? $\endgroup$
    – Reader 123
    Commented Sep 13, 2021 at 20:38
  • $\begingroup$ That's the one! Now do you know how that lambda is used to do the transformation? (Lambda is a Greek letter that looks like $\lambda$.) $\endgroup$
    – Dave
    Commented Sep 13, 2021 at 20:43

1 Answer 1

0
$\begingroup$

You can use the predict() function from the bestNormalize class to get the corresponding point in the transformed dataset.

Let's demonstrate with a known distribution (Gamma) and transform it to standard normal distribution. The following figure shows that the cumulative probabilities on the transformed data computed with pnorm() match with those computed with pgamma() from the original data, as expected.

library(bestNormalize)
x <- sort(rgamma(100, 1, 1))
BN_obj <- bestNormalize(x)
x.t <- BN_obj$x.t #predict(BN_obj, x)
plot(pgamma(x,1,1), pch=19, xlab='index', ylab='probability')
points(pnorm(x.t), pch=19, col='red')
legend('top', c("P(X<=x)", "P(Xt<=xt)"), col = c(1, 2),
       text.col = "green4", pch = c(19, 19),
       merge = TRUE, bg = "gray90")

enter image description here

Now, let's say we have a new data point x_test from the original dataset and we want to compute the cumulative probability using pnorm() from the transformed dataset.

  • You can do it with predict() to apply the transformation to a new test datapoint (taken from the original distribution)

  • Then use pnorm() to compute the cumulative probability.

  • It will be approximately same as the cumulative probability computed from the original Gamma distribution, as shown in the next code snippet.

    x_test <- 1.2345
    x.t_test <- predict(BN_obj, newdata = x_test)
    pnorm(x.t_test)
    # [1] 0.6891166
    pgamma(x_test, 1, 1)
    # [1] 0.7090198
    all.equal(pnorm(x.t_test), pgamma(x_test, 1, 1), tolerance=0.03)
    # TRUE
    
$\endgroup$
1
  • 1
    $\begingroup$ Brilliant, thank you, @Sandipan Dey! I have just tried it and it has worked. Appreciate the help! $\endgroup$
    – Reader 123
    Commented Sep 13, 2021 at 21:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.