2
$\begingroup$

Suppose I have some sample $X$ drawn from some unknown multivariate distribution $F(A,B)$, and I want to test the null hypothesis that a particular point $x$ was drawn from $F$.

Would it be legitimate to fit the PDF for $F$ using kernel density estimation, then evaluate it at $x$ and take this as my p-value?

Likewise, if I'm interested in the conditional probability $p(x~|~A=a)$, could I basically use the same approach, but just evaluate the PDF conditioned on $A = a$?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

A p-value is not the value of a pdf, but an actual probability.

Wikipedia has it right:

In statistical significance testing the p-value is the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true.

That is, a p-value might be found from an appropriately defined cdf, rather than a pdf.

Let's now consider - assuming we have a sample from the right cdf - whether a kernel density estimate computed on that sample is appropriate for computing p-values.

There are a number of relevant points I can make:

i) the estimate of the CDF obtained from a kernel density estimate is smoother than the original ECDF; indeed smoothness (of the PDF estimate) is the point of kernel density estimates

ii) the kernel density estimate has larger variance than the thing it estimates; it's not an unbiased estimate of the PDF. The CDF estimate will therefore also have bias.

iii) kernel density estimates are "optimized" for estimating PDFs, not CDFs. If you want an good smooth estimate of a CDF, you may be better directing effort to optimizing desirable properties of that.

iv) in the univariate case, often the preference seems to be to use the sampled values themselves to estimate empirical quantiles (though it is sometimes done); I don't know that this appearance of a preference carries over to multivariate cases.

In aummary: Yes, it's legitimate in the sense that - done correctly - it might provide reasonable estimates of p-values. It will be biased (tend to give too high a p-value in the tails and correspondingly too low in the middle). You should take care to try to optimize your smoothing to the purpose you are putting it to.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for your response. Do you still think that kernel density estimation could give me an "appropriately defined cdf" to use in this way? $\endgroup$
    – ali_m
    Commented Apr 17, 2013 at 10:26
  • $\begingroup$ see my update about smoothing and estimates of cdfs. Note that the point about 'appropriately defined' relates to your particular null and alternative hypothesis and the choice of test statistic. $\endgroup$
    – Glen_b
    Commented Apr 17, 2013 at 23:06
  • $\begingroup$ I see. The points that I'm interested in evaluating tend to fall towards the extremes where I have relatively few actual samples in my empirical distribution. I figured that I might be able to make a 'better' estimate of the true CDF of the underlying distribution than I could get by computing the ECDF from the ranks of my samples by assuming that the PDF was smooth and applying KDE. In any case, I'm somewhat reassured that my estimated CDF ought to tend to overestimate p-values in the tails, since this will just make my hypothesis test overly conservative and I care more about type I errors. $\endgroup$
    – ali_m
    Commented Apr 18, 2013 at 2:04
  • $\begingroup$ If you're in the tail yes, it could make sense to smooth, with appropriate caveats. $\endgroup$
    – Glen_b
    Commented Apr 18, 2013 at 2:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.