4
$\begingroup$

Say I have a data set composed of $N$ objects. Each object has a given number of measured values attached to it (three in this case):

$x_1[a_1, b_1, c_1], x_2[a_2, b_2, c_2], ..., x_N[a_N, b_N, c_N]$

meaning I have measured the properties $[a_i, b_i, c_i]$ for each $x$ object. The measurement space is thus the space determined by the variables $[a, b, c]$, where each $x$ object is represented by a point in it.

More graphically: I have $N$ objects scattered in a 3D space.

What I need is a way to determine the probability (or likelihood?, is there a difference?) of a new object $y[a_y, b_y, c_y]$ of belonging to this cloud of objects This probability calculated for any $x$ object will of course be very close to 1.

Is this feasible?

Add 1

To address AdamO's question: the object $y$ belongs to a set composed of $M$ mixed objects (with $M$ > $N$). This means that some objects in this set will have a high probability of belonging to the first data set ($N$) and others will have a lower probability. I'm actually interested in these low probability objects.

I can also come up with up to 3 more data sets of $N1$, $N2$, and $N3$ objects, all of them having the same global properties as those in the $N$ data set. Ie: an object in $M$ that has a low probability in $N$ will also have low probabilities when compared with $N1$, $N2$ and $N3$ (and viceversa: objects in $M$ of high probabilities of belonging to $N$ will also display high probabilities of belonging to $N1$, $N2$ and $N3$).

Add 2

According to the answer given in the question Interpretation/use of kernel density I can not derive the probability of a new object belonging to the set that generated the $kde/pdf$ (assuming I would even be able to resolve the equation for a non-unimodal $pdf$) because I have to make the a priori assumption that that new object was generated by the same process that generated the data set from which I obtained the $kde$. Could someone confirm this please?

|cite|improve this question
$\endgroup$
  • $\begingroup$ Do you also have a random sample of objects that don't belong to the $x_i[a_i, b_i, c_i]$, $i \leq n$ group? $\endgroup$ – AdamO Jul 3 '13 at 19:17
  • $\begingroup$ @AdamO not exactly, please see expanded question. $\endgroup$ – Gabriel Jul 3 '13 at 19:36
  • $\begingroup$ so basically you're looking for a density based clustering algorithm? Do you have an expectation for how many clusters there ought to be, or is this completely unsupervised? $\endgroup$ – David Marx Jul 3 '13 at 20:47
  • 1
    $\begingroup$ The question in your Add2 is actually somewhat unrelated. The question has to do with the interpretation of the $y$ axis and whether it has a direct probability interpretation. That obviously is incorrect. However, integrating a KDE over distinct bounds will give you interpretable probability estimates. $\endgroup$ – AdamO Jul 7 '13 at 15:17
  • $\begingroup$ But the answer accepted mentions precisely the area under the curve (ie: the integral of the $pdf$) and it still says it's not valid to interpret that as a probability of a point belonging to the set that generated that $pdf$. I think the last sentence pretty much says it all: The kernel density estimate doesn't say anything about the probability a new value was generated by the same process. What am I missing here? Thank you for your patience! $\endgroup$ – Gabriel Jul 7 '13 at 16:03
2
$\begingroup$

The first question to address is: what's your distance metric? If you're comfortable with Euclidean space, by all means use that. However, you may want to transform these data onto an orthogonal basis using some kind of SVD and that can be done easily with any statistical software.

Given these data have been transformed into a suitable domain, you can estimate a probability density for these data using some kind of parametric or nonparametric estimation. Roughly normal data are adept to estimation via maximum likelihood, but density smoothers like the boxcar, or (better) a radial basis kernal smoother, will give you an estimate of the probability density ($\hat{f}$) over your domain ($\Omega$).

With these in place, we can evaluate a new observation in terms of its probability from having originated from that distribution. With a new observation taking values $x, y, z$, integrate the probability density over values of the support for which the density is less than the one you observed. This is well behaved for unimodal distributions. That is,

$$\mathcal{F}(x, y, z) = 1-\iiint_{r, s, t :\hat{f}(r, s, t) < \hat{f}(x, y, z)} \hat{f}(r,s,t)\,dr\,ds\,dt $$

This has a direct interpretation like a p-value (very roughly and blending Bayesian / Frequentist ideas): with this point, assuming it is generating from a known distribution (\hat{f}), what's the probability of observing another point as improbable or more improbable given it comes from this distribution? If this value is sufficiently small, we would rule that it is unlikely to have originated from the same distribution, though there is a chance that results in a Type I error.

curve(dnorm, from=-5, to=2)

points(x=-1.8, y=dnorm(-1.8), col='red', pch=20)

polygon(
  x=c(-5, seq(-5, -1.8, by=.1), -1.8),
  y=c(0, dnorm(seq(-5, -1.8, by=.1)), 0),
  col='black'
)

polygon(
  x=c(1.8, seq(1.8, 2, by=.1), 2),
  y=c(0, dnorm(seq(1.8, 2, by=.1)), 0),
  col='black'
)

text(-1.8, dnorm(-1.8), paste("p(this observation | dist'n holds) 
= ", round(pnorm(-1.8)*2, 2)), pos=2)

enter image description here

|cite|improve this answer
$\endgroup$
  • $\begingroup$ I think I almost understand this answer (I have almost zero statistical training and what little I know is self-taught) So with this approach the hard part would be (assuming euclidean space is fine) obtaining the $\hat{f}$ probability density. Am I understanding correctly if I say that that $\hat{f}$ could be obtained through Duong's kernel density estimator (jstatsoft.org/v21/i07)? $\endgroup$ – Gabriel Jul 3 '13 at 21:28
  • $\begingroup$ Yes, although Duong is hardly worth being the namesake for the method. Density estimation has been approached several times especially since the advent of statistical computing. $\endgroup$ – AdamO Jul 3 '13 at 21:47
  • 3
    $\begingroup$ This solution appears to ignore all uncertainty in estimating $\hat{f}$ (which, for a kernel smoother, can be substantial). That changes the interpretation of the answer, which no longer can be viewed either as a probability nor as a p-value. How would you address this issue? $\endgroup$ – whuber♦ Jul 5 '13 at 15:47
  • 3
    $\begingroup$ @whuber +1 Yes, uncertainty in the estimate of the DF is a substantial caveat. For approximately Gaussian fields, estimating the joint density with ML would do substantially better. The question of "belonging to a set of objects" without the presence of a control or alternative comparison group is a question of consistency, i.e. how consistent is the observed point $y$ with the resampling prob dist-n of a fixed set of data. If $f$ were known, then the integral can be interpreted as a probability that a newly observed point having df $f$ belongs to any subset of $\Omega$ containing $y$ that... $\endgroup$ – AdamO Jul 7 '13 at 15:05
  • 1
    $\begingroup$ In 2D, you just need to parameterize connected curves in 2 space that determine lower and upper bounds. Think of an island and underwater regions on a map. $\endgroup$ – AdamO Jul 10 '13 at 17:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.