Probability of observing a measurement with error under a non-normal distribution

Suppose I have some kind of distribution which gives the probability of observing a given value (assume that this distribution is not normal).

Now, I have a certain observation and I want to calculate how likely it is to observe a value that is equally or more extreme under this distribution. So, I am thinking of doing this empirically by turning my observation into a quantile and calculating the probability empirically under the distribution of: quantile_distribution < quantile_observation and quantile_distribution > abs(1 - quantile_observation)

I'm not sure whether that is a valid way to do it, but for the second part I'm going to assume it is.

Now, here's where it gets a bit more complicated:

My observation has some sort of measurement error associated with it. I can model this error with a probability distribution (this time it is roughly normal). Given this spread, how would I calculate a p-value as above (or would it be a distribution of p-values, which I have never seen)?

I've been hacking something together where I take the p-value of observing the mean and the 95% CI values under the first distribution and then just take the maximum and report that, but obviously there has to be a better way to do it

• Basically what you have is a distribution that is the sum of 2 component distributions. If these were both normal, this would be trivial. Do you know what the other dist is? In general it would help if you could provide more details about your situation, your data & your goals; reading this blog post might help: How to ask a statistics question. – gung Mar 3 '13 at 18:03
• Okay, let me be more clear about the situation – user21504 Mar 3 '13 at 20:33

Call the variable that has the non-normal distribution $X_1$, call its cumulative density function $G(\cdot)$. Call the cutoff value $c$. In the no noise situation you want to know $P(X_1 \ge c) = 1-G(c)$.

Call the noisy cutoff value $X_2$. You say it is distributed approximately $N(c,\sigma^2)$. Call its cumulative density function $F(\cdot)$.

You want to know: $P(X_1 \ge X_2)=P(X_2 - X_1 \le 0)$. Define $Z=X_2-X_1$, and call its cumulative density function $H(\cdot)$. Then you want to know $H(0)$.

Easy way: Assuming you can draw samples from these distributions then use a Monte-Carlo approach. Take many draws of $(X_1, X_2)$. Then compute the proportion of those draws where $X_2-X_1 \le 0$.

Hard way: Assuming the noise is independent of $X_1$ you can use the result on the sum of independent random variables here: http://www.statlect.com/sumdst1.htm

It shows that: $H(0) = E[F(X_1)]$ (since you have a difference instead of a sum). Since you know $F$ is normally distributed it follows that:

$$H(0) = E\left[ {\frac{1}{2}\left( {1 + erf\left( {\frac{{ X_1 }}{{\sqrt {2\sigma ^2 } }}} \right)} \right)} \right]$$

To compute that you'll need to use the pdf of your non-normal variable $X_1$.