Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Given, $p_{ij}=\frac{|A_i|+|A_j|}{|A_i\cup A_j|}$ for sets $A_i$,$A_j$ $\forall i\not=j ,\forall i \in \left \{ 1,2...n \right \} $ and given the fact that $|A_i \cap A_j|>0$ for all $A_i,A_j$: it is clear that $p_{ij}$ > 1.
$p_{ij}\in\mathbb{R}^+, \forall i,j$ and $|A_i|\in\mathbb{Z}^+$ are random variables (observed cardinalities over finite sets are random variables). The $p_{ij}'s$ are calculated from an observed sample of $|A_i|$,$|A_j|$ and the cardinalities $|A_i \cap A_j|, |A_i \cup A_j|, \forall i,j$, are also known.

Properties: Since, $|A_i|+|A_j| >|A_i|\cup |A_j|$ ,

$$\frac{|A_i|+|A_j|+\epsilon}{|A_i\cup A_j|+\epsilon}<\frac{|A_i|+|A_j|}{|A_i\cup A_j|};\forall \epsilon>0$$

i.e, for a fixed $|A_i|$ greater values of $|A_j|$ will always lead to smaller $p_{ij}'s$ and smaller values of $|A_j|$ will lead to greater $p_{ij}'s$ as for a fixed $|A_i|$, the numerator and the denominator increase or decrease by the same amount.

Also,the $max(p_{\mathbb{.}j})$ for different columns $j$ can be different.

Applied Statistical Context(a): The larger the $p_{ij}'s$ in a column $p$ compared to column $q$, makes us believe that the set $p$ is better than set $q$ purely in an applied scenario when I say 'better'.

Applied Statistical Context(b): These are slightly similar to this applied example: I get 96/100 in a math test, and my brother gets 450/500. Who did better? There are ways to go about this if the observed proportions are considered binomial with sample sizes 100, 500 respectively. Here, our sample size in the denominator is fixed, unlike my example where the denominator is a random variable.

Now in order to analyze the columns of the matrix $P$, with entries $p_{ij}$, a primary issue is that the range of the values in different columns are not the same. How can I transform/(also may be normalize) the columns such that they are all within a fixed range [l,u] and also the entries in different columns need to taken to be account, instead of scaling individual columns. This may require tweaking my definition of $p_{ij}$ or might require some transformations over the columns taken together -or both. Also, feel free to make distributional assumptions over $|A_i|$-if you would prefer that direction.

share|improve this question
I find this question a bit vaguely worded and ambiguous in a couple of places. Pinning down precise meanings for terms like "better", "comparable", etc. may guide you to a more easily answerable question. You also talk of the observations being $|A_i|$, but then presumably you also (somehow) observe $|A_i \cup A_j|$ (for all pairs $i,j$?). The more detail you can provide here, the better. That would also help better substantiate your bounty statement. At any rate, have you looked at the Sinkhorn-Knopp (aka iterative proportional fitting) algorithm? – cardinal Mar 23 '12 at 16:26

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.