2
$\begingroup$

I've been applying the probability integral transform as shown here:

https://math.stackexchange.com/questions/592076/mapping-cdfs-to-each-other?noredirect=1&lq=1

I want to standardise the data that is wiebull distibuted by mapping it to standard normal:

The original histogram is shown below:

Original Histogram

When I transform to the standard Normal I get the histogram shown below, which doesn't look very Normal. I don't want the big gap I want the transformed histogram to look as much like a standard normal as possible. Is there anything else I can do?

Transformed Data

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ You need to explain the situation in more detail. Your data appear to be discrete -- are your data counts/scaled counts, or are they rounded/truncated measurements? What is the basis of your claim that your data are Weibull-distributed? Why are you trying to transform to normality? $\endgroup$
    – Glen_b
    Oct 5, 2016 at 9:15
  • $\begingroup$ The data is not discrete, (even if it was I's approximate using a continuous distribution so that I could apply the probability integral transform). The min value of 0.1 and max 23. It corresponds to a time series of financial data (bid ask spreads) these are not truncated per se but as they refer to tick data there is some discreteness to the moves. Weibull was best fit but I've tried several candidate distributions and they all produce a mapping qualitatively similar to this. I'm trying to convert to normailty as it will be input to a neural network and I need to standardize the inputs. $\endgroup$
    – Baz
    Oct 5, 2016 at 10:56
  • 1
    $\begingroup$ Tick data are discrete. The discreteness in the tick data is why it looks weird on the left hand side after you transform it. "Weibull is the best fit" does not mean it's Weibull and you shouldn't assert something to be Weibull when it's just a distribution you fitted from among a short list -- that's just a model you decided to use. Weibull may be suitable as an approximate model for some purposes but its lack of fit at the left end becomes apparent when you transform it. $\endgroup$
    – Glen_b
    Oct 5, 2016 at 11:46
  • 1
    $\begingroup$ to clarify -- while it often makes sense to approximate tick data with a continuous distribution, the discreteness down at the small values means that under transformation that stretches values near 0, the actual discreteness becomes what drives the behavior of the result (you can see the discreteness in your transformed values -- that's why it looks wrong). Your continuous model doesn't alter the fact that all the 0.1 values move together and end up in the same place. $\endgroup$
    – Glen_b
    Oct 5, 2016 at 22:05

1 Answer 1

Reset to default
1
$\begingroup$

Data fed to a neural network is standardized (vague term) s.t the magnitude of the data is not too large because too nominally large values leads to slow training (many steps to get right size of weights) and possibly exploding gradients (too big steps at the same time) leading to NaN, killing the ANN. It's also typically very nice to have data that is centered around 0 depending on the activation function. 'Standardizing' as $(x-\bar{x})/s$ is one way of doing this. If data becomes approximately normal it's nice but not necessary. (This won't happen anyway as the left tail won't grow further than the estimated mean as $W\geq 0$)

Point is that you don't need to spoonfeed an ANN, it'll eat almost anything. Besides you use it to learn the distribution of the data so I wouldn't worry too much.

Your question is vague, but if data is approximately Weibull$(\alpha,\beta)$ you can easily estimate the parameters and invert it to approximately uniform distribution since $W=\alpha(-\log(U))^{1/\beta}$. This can then be mapped as $(2U-1)\in [-1,1]$ if you really worry about it. Much less headache is to transform it as $\log(W+\epsilon)$ if you really want it to behave normalish. The standardized sample mean of $log(W)$ and any $(-\infty,\infty)$ random var is approximately standard normal but not $log(W)$ itself.

ps. I think the whole discussion about discretization is OT. First of all a discrete weibull with high $\alpha$ i.e resolution is almost perfectly fit with a continuous Weibull. I included a discretized distribution below

Continuous Weibull

Continuous data

Discretized (rounded up) Weibull

discrete data

a=100
b=1.1
n=100000
u = rnorm(n)

#png(filename="answer_discrete.png")

par(mfrow=c(2,2))
w =(a*(-log(u))^(1/b))
w = ceiling(w)

w_std = (w-mean(w,na.rm=T))/sd(w,na.rm=T)
hist(w,100,main='w',xlab='')

hist(w_std,100,main='standardized w',xlab='')

w = log(w)
w_std = (w-mean(w,na.rm=T))/sd(w,na.rm=T)
hist(w,100,main='log(w)',xlab='')
hist(w_std,100,main='standardized log(w)',xlab='')

#dev.off()

edit: had W=alog(u)^(1/b), should be a(-log(u))^(1/b). Updated graphs/code to reflect this

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for this very helpful. The data I am modelling comes from a time series of bid ask spreads based on tick data so there is an inherent discreteness which gives the similar ugly results when you used rounded data. In such a circumstance what is the best way to get round this discreteness problem. I thought by fitting it to a continuous weibull distribution as an approximation to the true discrete distribution, would resolve the problem even if it didn't give the best fit. What is the best way to handle data that is inherently discrete? $\endgroup$
    – Baz
    Oct 7, 2016 at 16:53
  • 1
    $\begingroup$ It really depends on what you want to do. To train a neural network it'll eat discrete or continuous values like there's no difference. Are you interested in the parameters $\alpha$ and $\beta$? At this resolution contin. and discr. are almost equivalent par $W=0$ but I'd go with the discrete Weibull. Multiple packages implements it. Can't find? Fit a contin. Weibull but add some uniform noise (0,1) of the magnitude of the discrete timesteps to avoid $W=0$. This is definitely a ugly hack but as $\beta>1$ by inspection this won't matter too much. Also check tinyurl.com/zgzgvuq. $\endgroup$
    – ragulpr
    Oct 8, 2016 at 13:12
  • 1
    $\begingroup$ Also if you want to do some kind of Weibull-Neural-network-magic for timeseries that's what I'm currently working on, i.e using a weibull kernel to predict $\alpha$ and $\beta$ to get a distribution over the future so I'd be happy to talk about that $\endgroup$
    – ragulpr
    Oct 8, 2016 at 13:17
  • $\begingroup$ Could be interesting for me too, how do I get in touch? $\endgroup$
    – Baz
    Oct 9, 2016 at 16:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.