7
$\begingroup$

I am calculating rates, which can take any value between 0 and 1. Can it be normally distributed even though the domain is not the real numbers?


Normal distribution fit to the means of the lapses (bootstrapped data)

Thank you very much for the answers, here I represent the means of the data which are fitted a normal distribution on. I created something like 1000 means of the data using bootstrapping.

Raw data

As for the raw data, it is indeed heavily skewed with a large positive skewness value. Based on your answers, the normality for t-test can't be assumed 100%. Instead of t-tests, I'm trying to calculate confidence intervals. I have one confidence interval for the prediction using bootstrapping, although I'm not a 100% sure this is the correct way. I'm comparing 4 predictive models to decide what gives the best results. Individual predicted rates are grouped by the age of the policy and taken their average, so the predictions are for example: for the age=4 the rate = 4.2%. I want to use another method for the CI, namely the Chebyshev's inequality. But for this I need to fit a distribution to the data. I already tried weibull, beta, gamma but none of them seem to work.

EDIT: The model I created predicts individual rates and I take the average of these rate to get the mean rate for a group. That mean has to be estimated correctly, also be assigned a CI to it. I figured that if I perform a t-test on every group between the model predictions and the actual values that need to be predicted (test dataset) and I get not significant p-values, then the model is good. I needed this information regarding the possible normality of the values because of the t-test.

Thank you very much for all the information you've give me so far! You are great!

$\endgroup$
  • 12
    $\begingroup$ Only approximately. Alternatively if you check out say the beta distribution you will find that this respects the bounds yet can be close to symmetric. $\endgroup$ – Nick Cox Jan 28 at 11:30
  • 2
    $\begingroup$ ... can be exactly symmetric too! $\endgroup$ – Nick Cox Jan 28 at 11:42
  • 1
    $\begingroup$ In many cases rates do not exhibit distributions that can be well approximated by a Normal distribution, especially when many of the rates are extreme (close to $0$ or $1$), so you might be looking in the wrong place if you're trying to develop a probability model for your calculated rates. $\endgroup$ – whuber Jan 28 at 14:12
  • 1
    $\begingroup$ It depends a lot on the situation. It may very well be the case that your rates can be approximated with a normal distribution (I assume that this approximation, instead of exact equivalence is what you are aiming for). When you are computing rates then often you are computing counts. These counts may be binomial distributed, which can be well approximated with a normal distribution if the number is sufficiently large.... $\endgroup$ – Sextus Empiricus Jan 29 at 0:02
  • 3
    $\begingroup$ what are you trying to achieve? why it is important whether (and by how much) your data is "normally distributed on [0,1]" $\endgroup$ – aaaaa says reinstate Monica Jan 29 at 19:59
19
$\begingroup$

No, it cannot. At least if you by "distributed as" implies exactly. The range of the normal distribution extends from minus to plus infinity. As a practical matter, if the variance is sufficiently small, say on the order of $ (0.1)^2 $, then a variable constrained to $(0,1)$ can be approximately normally distributed.

$\endgroup$
  • 1
    $\begingroup$ Lapse rate can mean many things but none that I know of has an upper bound of 1 (those I know about have units of measurement, so even if bounded the upper bound depends on a convention about units.) $\endgroup$ – Nick Cox Jan 28 at 12:28
  • 2
    $\begingroup$ Fine; it's really a probability. I wouldn't use a normal here at all, even for means. $\endgroup$ – Nick Cox Jan 28 at 12:35
  • 4
    $\begingroup$ You're telling us that the mean is very close to the boundary. That's always dangerous. I can't but prefer to work on a transformed scale or use a non-normal distribution as reference if I had similar data. Assuming that data are as you prefer has many advantages, but it can be wishful thinking. Your data are, I guess, not only too large to show us but also likely to be confidential or sensitive, but I would love to see a quantile plot. $\endgroup$ – Nick Cox Jan 28 at 13:27
  • 1
    $\begingroup$ @Alexis I am better at numbers/images than words. So, when we correct the logic (it is not exactly normal distributed) and the language (a single mean doesn't have a distribution. We can not say an observation is distributed) then it becomes: "I assume the mean to be sampled from a distribution that can be approximated with a normal distribution" or shorter "I assume the mean can be modelled/approximated with a normal distribution". $\endgroup$ – Sextus Empiricus Jan 30 at 9:35
  • 1
    $\begingroup$ @fabiob If you have a bunch of Bernoulli distributed outcomes like: $$X = 0.004, 0.02, 0.02, 0.02, 0.02, 0.004$$ then their transform will be just as well a Bernoulli distributed but only with different values $$log(X/(1-X)) = 5.52, -3.89, -3.89, -3.89, -3.89, -5.52$$ when you are doing logistic regression then often you do not transform the outcome variable, but instead you transform the expected mean. $\endgroup$ – Sextus Empiricus Jan 30 at 12:58
7
$\begingroup$

The answer to your literal question is "no", but the larger implicit question of how you should model your data is more complicated. As Jim says, a truncated normal model is one option. You can also look into converting your probabilities to log odds, which will range from $-\infty$ to $\infty$, or the Beta distribution as Nick Cox mentions.

The Central Limit Theorem does in some sense apply to your data, but the CLT just says that the data goes to the normal distribution in the limiting case, it doesn't say that any particular distribution for finite sample size is normally distributed. That is, for any level of precision, there is some sample size for which the distribution is normal within that level of precision, but that doesn't mean that you have enough sample size for it to be normal to the level of precision needed.

You mention in comments that the probabilities are small, which likely means the data is skewed. The more skewed data is, the larger a sample size is needed to get to a particular level of precision using the CLT. So you might want to look into approximating with a skewed distribution, such as Poisson. Depending on the data, you could converge to such a distribution faster than to normal.

In the worse case scenario, you can probably use Chebyshev bounds.

$\endgroup$
6
$\begingroup$

By definition the normal distribution has support $(-\infty, \infty)$.

You may want to look into the truncated normal distibution. It can have bounded support $[a,b]$. Quoting from its wiki:

[...] the truncated normal distribution is the probability distribution derived from that of a normally distributed random variable by bounding the random variable from either below or above (or both).

$\endgroup$
2
$\begingroup$

Many situations are not exactly normal distributed. Possibly most practical situations might be not be truly normal distributed (when we model human length or weight by a normal distribution, does that mean that we consider negative values?).

The normal distribution is a distribution of many numbers. When you have a sum of many effects/variables then the distribution will follow approximately the normal distribution. The first application of the normal distribution (or something that looks like it) dates back to deMoivre who used it as a model to approximate a binomial distribution (which does not have infinite support), which can be considered as a sum of many Bernouilli distributed variables.

The question for you is whether your particular situation allows the use of an approximation with the normal distribution. You have mentioned in the comments a mean/sum of 400k samples, that sounds very much like a (approximately) normal distributed variable (although, depending on your goals, you might still wish to investigate more than just the mean of your sample, and gather more information from the distribution of your samples which is likely not normally distributed, since we are speaking of few, individual, numbers).

Below is an image of a histogram (and normal approximation) of $X/400000$ with $X \sim Binom(n=400000,p=0.04)$. This variable ranges from 0 to 1.

example

$\endgroup$
0
$\begingroup$

Strictly speaking, a variable defined on a finite interval cannot be normally distributed. However, as mentioned previously it can be approximately so.

In addition, in some cases it can be transformed to a normally distributed variable. For example, the Pearson correlation coefficient between two independent variables, which is restricted to a finite interval ($-1\le r\le1$), can be transformed to an approximately normally distributed variable $z$ using the Fisher transformation: $$z = {1\over2}\ln{1+r\over1-r}$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.