3
$\begingroup$

Say that we have two processes $a$ and $b$ that generate numbers. Then say that the random variables $A$ and $B$ take values in the set of numbers that those processes generate, respectively. So $f_A$ and $f_B$ are their probability density functions, respectively.

Now, suppose that we plot those PDFs $f_A$ and $f_B$, and find that the intersection area between them is $0.6$. In other words (as Aksakal and Tim put it): $$\int_x \text{min}\big(f_A(x), f_B(x)\big)\,\,\text{d}x = 0.6$$

Is $0.6$ a probability? Is it the probability that the processes $a$ and $b$ might generate numbers that are statistically indistinguishable?

Or, do I need to normalize $0.6$ somehow in order to make it into a probability?

To further clarify:

  • The question is not:

    "What is the probability that the processes are the same?". In fact, we can assume that the processes are necessarily different.

  • The question is rather:

    "Suppose that a randomly sampled number, from either one of the processes, was handed to you, without telling you the process that generated the number. Then, you were asked to predict the process that generated that number. Is this intersection area, $0.6$, the probability that the number that is given to you is statistically uninformative to you with respect to the mother process that generated it?"

$\endgroup$
  • $\begingroup$ No but its integral is. $\endgroup$ – Michael R. Chernick Apr 3 '17 at 14:24
  • 4
    $\begingroup$ @Michael By definition, the area above a measurable set of real numbers and below the graph of a distribution function must also have units of probability. Thus a geometric combination of such areas--such as the intersection of two such areas--must also be in probability units. The only issue worthy of consideration is whether its value could be interpreted as a probability--and if so, what event (if any) it is the probability of. You are confusing the situation by confounding that area with its "integral"--whatever that might be. $\endgroup$ – whuber Apr 3 '17 at 14:56
  • 1
    $\begingroup$ Could you please clarify what you mean by intersection area? In my answer below I interpret it to mean the absolute area between the densities, that is $\int | f(x)-g(x)|\; dx$. Is that what you have in mind, or something else? $\endgroup$ – kjetil b halvorsen Apr 3 '17 at 15:14
  • 4
    $\begingroup$ @caveman do you mean the area of the overlap in two density-curves? Given there's some ambiguity -- and it appears different readers are carrying different understandings -- perhaps you could draw an example of what you mean. [If that conveys what you mean well enough, feel free to use the image from this comment] $\endgroup$ – Glen_b -Reinstate Monica Apr 4 '17 at 1:12
  • 2
    $\begingroup$ It looks like total probability of misclassification in two-group discriminant analysis. $\endgroup$ – L.V.Rao Apr 4 '17 at 13:36
2
$\begingroup$

If I understood your question correctly, then this is what you're looking for: $$\int_x\min [f_A(x),f_B(x)] dx$$

Is it the probability that the processes a and b might generate numbers that are statistically indistinguishable?

No. Only if this quantity is equal to 1, you'll get two processes statistically indistinguishable. They'll be like two realizations of the same process.

UPDATE Say you have 2 rows of 10 bins each. You fill the first row with 100 red balls, and the second row with 100 blue balls. Next, you look at the first two bins, and pick either red or blue balls, whichever is the fewest. Then you do the same with a next pair of bins, and so on until all 10 pairs are exhausted. How many balls out of total 200 you collected? The quantity that you're looking at answers this question.

Overall, this type of quantity seems like something you might encounter in game theory or stochastic programming and dynamic decision process

$\endgroup$
  • $\begingroup$ Thank you. But I think that there is some chance that maybe you misunderstood me. So just to clarify: I think what I said is not "prob. that 2 processes the same?". What I said is: assumption 1: processes are different, but question: if you see a randomly sampled number that was generated by one of those processes, except that you don't know which process, then can we use this intersection area as the probability that the processes could generate numbers are statistically difficult to be correctly attributed to their mother process? $\endgroup$ – caveman Apr 12 '17 at 2:54
  • $\begingroup$ Of possible interest: This is essentially a continuous version of the "histogram intersection kernel" commonly used in computer vision. (a Mercer kernel; commonly robust-ified using multiple scales) $\endgroup$ – GeoMatt22 Apr 12 '17 at 5:33
  • 1
    $\begingroup$ @caveman for that look at kolmogorov smirnov statistic. It's a similar measure but on cdf not pdf. $\endgroup$ – Aksakal Apr 12 '17 at 11:39
2
$\begingroup$

No, it is not, it is twice a probability!

Let $P$ and $Q$ be probability measures with corresponding densities $f$, $g$. Then the variation distance between $P$ and $Q$ is $$ \| P-Q \| = \max_A |P(A) -Q(A) | $$ so it measures the absolute difference in probability for an event that maximizes that difference.

Then one can show that (*) $$ \| P-Q \| =\frac12 \int |f(x)-g(x)| \; dx = \frac12 \max_{\| h \| \le 1 } | P(h)-Q(h)| $$ where $h$ is a real-valued function satisfying $|h(x) \le 1$ and $P(h), Q(h)$ denotes the expectation of $h$ (under that measure). Paul Switzer suggested the following interpretation: You are given a single observation, from $P$ or from $Q$, each with probability $\frac12$. You have to guess from which distribution it came. Your probability of being right is then $\frac12 + \frac12 \|P-Q\|$.

Despite requests, the OP did'nt clarify his interpretation in the question. @Aksakal offered another interpretation, $\int \min(f(x),g(x))\; dx$. But the variational norm can be used to give an answer in that case too. First, note that $$ f(x) + g(x) = \min(f(x),g(x)) + \max(f(x),g(x)) $$ and $$ \max(f(x),g(x)) = \min(f(x), g(x)) + \mid f(x)-g(x) \mid $$ Now, integrating this two equations and doing some algebra gives $$ \int \min(f(x),g(x)) \; dx = 1 - \frac12\int\mid f(x)-g(x)\mid \; dx= 1-\|P-Q\| $$ so, indeed, the variational norm gives an interpretation also in this case.

(*) First, the factor $\frac12$ is easy to understand, since $\int |f(x)-g(x)| \; dx$ has an infimum of zero (when $f=g$) and a supremum of 2 (when $f$ and $g$ have disjoint supports. Now a proof: There is a proof in a comment at this post: https://math.stackexchange.com/questions/69166/understanding-the-relationship-of-the-l1-norm-to-the-total-variation-distance

$\endgroup$
  • 2
    $\begingroup$ I suspect your interpretation of this question might not be what was intended by the phrase "intersection area." Many would understand that to be the area of the intersection of the two areas beneath each graph, whereas you seem to interpret it as the absolute area of the region between the graphs. Before you go to the effort of writing up a proof of anything, you ought to clear this up, lest it all be for nought. $\endgroup$ – whuber Apr 3 '17 at 15:06
  • $\begingroup$ @whuber: thanks, nevermind, the variational norm gives an interpretation also in case of Aksakal's interpretation. Whas that what you had in mind? $\endgroup$ – kjetil b halvorsen Apr 5 '17 at 7:10
  • 2
    $\begingroup$ My initial interpretation was that of Aksakal, but your post showed me the possibility of other interpretations. Like you and @glen_b, I am awaiting clarification from the OP. $\endgroup$ – whuber Apr 5 '17 at 14:03
1
$\begingroup$

If by "intersection" you mean $f(x) = \min\{f_A(x), f_B(x)\}$, then your function could be paraphrased, "among the possible generators, what is the lowest probability that this event will occur?" In general, this will not be a probability distribution.

This is easier to see in the discrete case. For example, say we have two unfair coins, $A$ and $B$, with chances for landing heads or tails given follows: $$f_A(x) = \begin{cases} 0.9 & x = heads \\ 0.1 & x = tails \\ 0 & otherwise \end{cases} \\ f_B(x)= \begin{cases} 0.1 & x = heads \\ 0.9 & x = tails \\ 0 & otherwise \end{cases}$$ Then your intersection function will give $$\begin{align} f(heads) & = \min\{f_A(heads), f_B(heads) \} \\ & = \min\{0.9, 0.1\} \\ & = 0.1 \end{align} $$ Similarly $f(tails) = 0.1$. Then, since $\sum_{x \in \{heads, tails\}} f(x) = 0.2 < 1$, we clearly don't have a probability distribution. Neither is it the probability that both $A$ and $B$ are "indistinguishable" $$\begin{align} P("indistinguishable") & = P(A = B) \\ &= \sum_{x \in \{heads, tails\}}P[f_A(x) \cap f_B(x)] \\ &=[f_A(heads) \cap f_B(heads)] +[f_A(tails) \cap f_B(tails)] \\ &= [0.9 \times 0.1] + [0.1 \times 0.9] = 0.18\end{align}$$

Intersecting PDFs is different from the intersecting the events $a$ and $b$. Then we talk about joint, marginal, and conditional probability distributions. An intersection of two events in the joint probability distribution for the events. The way to think about this, rather than intersecting two PDFs, you have to increase the dimensionality of your probability space to 3 dimensions: $x$ is values of $A$, $y$ is values of $B$ and $z$ is the joint probability that $P(A = x \cap B = y)$. If you project the PDF onto the $x-z$ plane, you should get back your marginal distribution $f_A$, and if you project it onto the $y-z$ plane, you get your $f_B$ marginal back.

When you talk about "statistically indistinguishable," it makes me also think of hypothesis testing. Frequently, hypothesis testing textbooks will show two distributions side by side, and one tail is highlighted. These graphics can be confusing, but it isn't taking the intersection of the distributions. One distribution is the null hypothesis ($H_0$) and the other is the alternative hypothesis ($H_A$). The pictures then usually highlight the tail of $H_0$, which are the unlikely events. When we observe these events, we say with a certain degree of statistical confidence that $H_0$ is not the generator for those events, and instead we favor $H_A$ as a more likely generator.

$\endgroup$
  • $\begingroup$ It is difficult to see how this will give a conditional probability: the area in question appears to be the area under the graph of $x\to \min(f_A(x),f_B(x))$. Could you please explain why that would be a conditional probability and indicate exactly what it is being conditioned on? $\endgroup$ – whuber Apr 3 '17 at 17:00
  • 1
    $\begingroup$ @whuber you're right... my talk on conditional is totally off-base. Thanks. $\endgroup$ – Tim Apr 3 '17 at 18:27
1
$\begingroup$

As others already noted, this is not a probability. However you can make probability out of it by scaling it appropriately.

Take

$$ f_{A \cap B}(x) = \min\{\, f_A(x),\, f_B(x) \,\} \\ f_{A \cup B}(x) = \max\{\, f_A(x),\, f_B(x) \,\} $$

then

$$ \frac{\int_x \,f_{A \cap B}(x) \;dx}{\int_x \,f_{A \cup B}(x) \;dx} $$

is probability, as it is the $A \cap B$ fraction of total area of $A \cup B$. However it is hard to find a meaningful interpretation of such probability (or possible variations of this problem).

Check the Mutual Information as probability thread for a similar solution of different problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.