Modeling success rate with gaussian distribution

Question

In many papers I see data representing a rate of success (i.e a number between 0 and 1) modeled as a gaussian. This is clearly a sin (the range of variation of the gaussian is all of R), but how bad is that sin? Under what assumptions would you say it is tolerable?

Answer 1

It depends on the data. While the normal distribution does span the real line do note that nearly 99% of the values are contained within 3 standard deviations of the mean. Thus, if the following conditions hold it may be a reasonable assumption:

(a) the data range is such that 99% of the data falls between [$\mu - 3\sigma,\mu+3\sigma]$

(b) the data is unimodal

(c) the data 'passes' other relevant tests for normality

Having said that some decision needs to be taken in the event that a draw from this distribution falls below 0 or above 1. Two ideas in such a situation:

(a) If draw < 0 set the draw to 0 and if draw > 1 set the draw to 1 or

(b) Model the distribution as truncated normal with the cut off points at 0 and 1.

Answer 2

It's usually a small sin. In nature, most phenomena can't realistically receive any value in R, but we model them as if they could.

The greater sin is to assume that the rate of success is shaped like a normal distribution if it isn't.

Answer 3

Are you entirely sure that they're using the normal distribution directly? It's very common to use transformed responses to model success rates, but this involves passing through a link function to move from a Gaussian random variable to a value in [0,1]. A commonly used transform is the probit one, which is just the inverse Gaussian CDF. (So you'd end up with something like $\Phi^{-1}(p) = X\beta + \sigma$, where $\Phi$ is the Gaussian CDF).

If you're actually using a normal distribution directly to model a result in [0,1], then it strikes me that the variance would have to be so small -- especially for p near 0 or 1 -- that you'd nearly always overfit the model.

Answer 4

Could you quote from "many papers" so that we would get some context? Between "Gaussian" and "number between 0 and 1" I see slight conflict as the draws from a Gaussian are not bounded. Maybe you meant p-values?