1
$\begingroup$

If you have a set $X$ of one dimensional samples sampled from some distribution, then it is possible to compute an estimate of the p-value for a test sample $x_i$ as follows (which is easy to implement) : $$p_{x_i} = \frac{|\{x_j \in X : x_j > x_i\}|}{|X|}$$

Now if you have d-dimensional samples (to simplify let's say 2 dimensions), is it possible to compute an estimate of the p-value with some similar method as above ?

Improve this question
$\endgroup$
3
  • $\begingroup$ Can you say a little more about the context (what you're trying to do), how the samples were generated, and what you want this value to measure? $\endgroup$
    – user20160
    Commented Feb 26, 2018 at 22:40
  • $\begingroup$ @user20160 Assume samples in $X$ are sampled from a one-dimensional Gaussian distribution with some mean and variance (just as an example). Now if you have a new set of samples $Z$, then the p-values for these samples would be uniformly distributed if they are sampled from the same distribution as previously (and vise versa), so we can just test for the uniformity of these p-value to test that. Now what if our samples are not one-dimensional ? How can we compute the p-values in that case ? $\endgroup$
    – eLearner
    Commented Feb 27, 2018 at 13:59
  • $\begingroup$ What hypothesis does this $p_ {x_i}$ relate to? What does the |...| mean, cardinality? What do you mean by 'p-value for a test sample'? What do you mean by 'estimate of the p-value'? $\endgroup$
    – Sextus Empiricus
    Commented Feb 27, 2018 at 18:03

1 Answer 1

Reset to default
0
$\begingroup$

I am not sure what is your end goal (consider elaborating more in your question), but I am skeptical p-values will bring you there. Also, I think you slightly misunderstand what p-values are. Lets assume each observation is real number (but the same applies to any other domain). To speak about p-value, after observing $n$ datapoints, you need the vector of observed values $Y_{obs} \in \mathbb{R}^n$, a random variable representing a vector of data that would be observed under the null hypothesis $Y_{null}$ and a test statistic $T : \mathbb{R}^n \rightarrow \mathbb{R}$. Than p-value is defined as

$p(Y_{rep}) = Prob[T(Y_{rep}) > T(Y_{null})]$

where the probability is with respect to the random variable $Y_{null}$ ($Y_{rep}$ is given). Further, p-value has to satisfy that if the null hypothesis is true ($Y_{rep}$ is drawn from the null distribution) than

$p(Y_{rep}) \sim Uniform(0,1)$

where both $Y_{rep}$ and $Y_{null}$ are treated as random variables with the same null distribution.

So you need to say what is your test statistic and what is your null distribution. If I understand you correctly you want to determine if a single new observation comes from the same distribution as a set of observations you saw previously. I believe this is very seldom an interesting question. If you want to do it, the approach you outlined ignores the uncertainty you have in the already observed samples. To take this uncertainty into account (without any assumptions on the underlying distribution), your task translates to the Kolmogorov-Smirnov test. In higher dimensions this task becomes even more difficult to do and you need more than a single observation to say anything with any degree of certainty. This answer has some pointers to comparing samples from multivariate distributions without any assumptions.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.