Suppose I have some kind of distribution which gives the probability of observing a given value (assume that this distribution is not normal).
Now, I have a certain observation and I want to calculate how likely it is to observe a value that is equally or more extreme under this distribution. So, I am thinking of doing this empirically by turning my observation into a quantile and calculating the probability empirically under the distribution of: quantile_distribution < quantile_observation and quantile_distribution > abs(1 - quantile_observation)
I'm not sure whether that is a valid way to do it, but for the second part I'm going to assume it is.
Now, here's where it gets a bit more complicated:
My observation has some sort of measurement error associated with it. I can model this error with a probability distribution (this time it is roughly normal). Given this spread, how would I calculate a p-value as above (or would it be a distribution of p-values, which I have never seen)?
I've been hacking something together where I take the p-value of observing the mean and the 95% CI values under the first distribution and then just take the maximum and report that, but obviously there has to be a better way to do it