Compare single observed value to simulated distribution I have a distribution of values that I have simulated for a null hypothesis data generating process. I have a single real-world observation that is wayyyy outside the percentiles of this distribution. Can I give a p-value for this single observation not being generated by the same data generating process that gave me the simulated distribution? I'd do something like a Z test, but the simulated distribution isn't quite Normal because it's cut off at 0. Intuitively I feel like I should be able to get a p value by direct comparison to the simulated distribution- because isn't that what other statistical tests do, say 'this is way at the edge of the distribution that would be generated by a certain process'?
 A: Let's assume (a) the $n$ values you generated are numbers and (b) you generated them independently.  Then your null hypothesis is that the real-world value is one more independently generated value.  The chance that it is the most extreme ("way outside") is the chance that it is either the first or the last in the sorted order.
Independence implies every value has a $2/(n+1)$ chance of being most extreme (or more).  Putting aside any concerns about HARKing, data snooping, and so-on, the p-value therefore is $2/(n+1).$

This corresponds to a standard way of estimating p-values using any simulation-based test, such as bootstrapping and permutation tests.
In the general setting the single real-world value is a statistic computed from a dataset and the simulation creates comparable datasets according to the null hypothesis and outputs their statistics.  In most of these applications the interest lies only in one extreme direction -- either very high or very low, but not both -- and the p-value is the proportion of all results (including the real-world value) equal to or more extreme than the real-world value.  Thus, if the real-world value is the $k^\text{th}$ from the extreme value in order out of all $n+1$ values, the associated p-value is $k/(n+1).$
A: It sounds like your simulated distribution is a truncated normal distribution.
If that is the case and you know the distribution parameters, you could use e.g. the ptruncnorm() function from the truncnorm package or ptnorm() from the crch package in R to calculate the p value for your observation. Use 0 and Inf as your lower (left) and upper (right) bounds, respectively.
Please note, however, that the p-value is the probability of obtaining values at least as extreme as the one you have under the assumption that the null hypothesis is true (i.e., that your observation indeed comes from a truncated normal distribution with the parameters that you have specified).
It is not the probability of "this single observation not being generated by the same data generating process that gave me the simulated distribution". In other words, it is not the probability that the null hypothesis is false. See https://en.wikipedia.org/wiki/Misuse_of_p-values.
In this case, the p-value calculation could be considered a Fischerian significance test, in which the p-value simply indicates the unusualness or "surprisingness" of your observation when compared to a chosen distribution. This is not to be confused with a Neyman-Pearson hypothesis test, where you decide in advance to automatically reject the null hypothesis if the test statistic falls outside a pre-specified threshold value α (often set to 0.05). The latter approach can be acceptable in some situations, but it is very important to be clear that the basis for rejecting the null hypothesis is arbitrary. See https://link.springer.com/chapter/10.1007/164_2019_286.
A: Thank you for this input! I found this paper answered my question: Graeme D. Ruxton and Markus Neuhauser (2013) Improving the reporting of P-values generated by randomization methods
It seems to say that for a large enough simulated sample, it is acceptable to say that p(equally extreme or more extreme) = number in simulated dataset(equally extreme or more extreme)/size of simulated dataset.
