Probability of observing a measurement with error under a non-normal distribution Suppose I have some kind of distribution which gives the probability of observing a given value (assume that this distribution is not normal).
Now, I have a certain observation and I want to calculate how likely it is to observe a value that is equally or more extreme under this distribution. So, I am thinking of doing this empirically by turning my observation into a quantile and calculating the probability empirically under the distribution of: quantile_distribution < quantile_observation and quantile_distribution > abs(1 - quantile_observation)
I'm not sure whether that is a valid way to do it, but for the second part I'm going to assume it is.
Now, here's where it gets a bit more complicated:
My observation has some sort of measurement error associated with it.  I can model this error with a probability distribution (this time it is roughly normal).  Given this spread, how would I calculate a p-value as above (or would it be a distribution of p-values, which I have never seen)?
I've been hacking something together where I take the p-value of observing the mean and the 95% CI values under the first distribution and then just take the maximum and report that, but obviously there has to be a better way to do it
 A: Call the variable that has the non-normal distribution $X_1$, call its cumulative density function $G(\cdot)$. Call the cutoff value $c$. In the no noise situation you want to know $P(X_1 \ge c) = 1-G(c)$.
Call the noisy cutoff value $X_2$.  You say it is distributed approximately $N(c,\sigma^2)$. Call its cumulative density function $F(\cdot)$.
You want to know: $P(X_1 \ge X_2)=P(X_2 - X_1 \le 0)$. Define $Z=X_2-X_1$, and call its cumulative density function $H(\cdot)$. Then you want to know $H(0)$.
Easy way:
Assuming you can draw samples from these distributions then use a Monte-Carlo approach. Take many draws of $(X_1, X_2)$.  Then compute the proportion of those draws where $X_2-X_1 \le 0$.
Hard way:
Assuming the noise is independent of $X_1$ you can use the result on the sum of independent random variables here: http://www.statlect.com/sumdst1.htm
It shows that:  $H(0) = E[F(X_1)]$ (since you have a difference instead of a sum).  Since you know $F$ is normally distributed it follows that: 
$$
H(0) = E\left[ {\frac{1}{2}\left( {1 + erf\left( {\frac{{ X_1 }}{{\sqrt {2\sigma ^2 } }}} \right)} \right)} \right]
$$
To compute that you'll need to use the pdf of your non-normal variable $X_1$.
