Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.

I want to see if a datapoint x should (or not) be assigned to a nearest component y using the following condition:

if ($d > T$) then {do not assign x to y}. With $d = distance(x,y)$ and $T = \bar{d}+\sigma$ where $\bar{d}$ is the mean distance of y to datapoints that were already assigned to it in the past, and $\sigma$ the associated stdev.

I want to replace this condition (i.e. ($d > T$)) by an equivalent expression which is expressed as a probability $p$. How can I do that ? I've tried for exemple $p = exp(\frac{-d^2}{2\sigma`^2}) / (\sigma`\sqrt{2\pi})$ but it does not behave like the first condition unless we manually find a good value for $\sigma`$

Note: it does not really matter if it is not a true probability (that sum to one).

share|improve this question
If you have any parameters at all in your probability calculation, they are going to have to be tuned somehow if you want good results, and if you don't have any parameters, you're not likely to get good results, except by great luck. Also, any probability calculation will have to take into account all the distances, otherwise you can't ensure the probabilities sum to one (note that even "set p=1 for the minimum distance component" takes all the distances into account when calculating the minimum.) – jbowman Jun 5 '12 at 13:44
@jbowman please be more clear, are you saying that I should keep $\sigma`$ and vary it manually (by increasing it) until I get a satisfying results ?! Note that the first condition works well, I just want to replace it with an equivalent probabilistic expression. – shn Jun 5 '12 at 14:06
Note that exp(−d$^2$/2σ`$^2$) is not a probability. You need to normalize by dividing it by (√2π)σ`. – Michael Chernick Jun 5 '12 at 14:21
@MichaelChernick this does not matter since we manually change $\sigma`$ until it behaves like the first condition. I don't know however if there is any way to "learn" the good value for $\sigma`$ – shn Jun 5 '12 at 14:34
What you have is only proportional to a probability density. It is not the normal probability density without the normalization. In any case the terminology is wrong. You were attempting to express the requirement in terms of the height of the probability density and not in terms of a probability. – Michael Chernick Jun 5 '12 at 14:40
show 11 more comments

1 Answer

If y is too far from x you should be able to define T independent of y. I wouldn't do it this way. But if you want to update sigma and T and then calculate P(d>T) you can and decide whether or not to keep y based on that probability.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.