3
$\begingroup$

Is there any reason to set $p_{ii} = 0$ or $q_{ii} = 0$ instead of 1 in t-SNE?

SNE

enter image description here

enter image description here $$ C = \sum_i KL(P_i || Q_i) = \sum_i \sum_j p_{i|j} \log \frac{p_{j|i}}{q_{j|i}} $$

Symmetric SNE

enter image description here

enter image description here $$ C = KL(P_i || Q_i) = \sum_i \sum_j p_{ij} \log \frac{p_{ji}}{q_{ji}} $$

The author of t-SNE, L. var der Maaten said below at t-SNE paper. "For nearby datapoints, $p_{j|i}$ is relatively high, whereas for widely separated datapoints, $p_{j|i}$ will be almost infinitesimal."

I think $P_{j|i}$ or $P_{ji}$ is the similarity function of the data points. Therefore if $j = i$, $p = 1$ is more reasonable than $p = 0$.

So, is there any reason to set $p_{ii} = 0$ or $q_{ii} = 0$ instead of 1 in t-SNE?

$\endgroup$
  • $\begingroup$ @utobi thanks for edit - I added some formula about Probability functions $\endgroup$ – Choung young jae Oct 31 '16 at 5:31
1
$\begingroup$

Citing the original article: (p. 2581, paragraph right below the equation 1)

Because we are only interested in modeling pairwise similarities, we set the value of $p_{i|i}$ to zero.

If that's a good reason, I'm not sure. But in practice I think it doesn't change anything since $p_{ii} = 0$ and $q_{ii} = 0$ or $p_{ii} = 1$ and $q_{ii} = 1$ both give the same value to the cost function $C$.

Intuitively I think you are right, It makes more sense to set $p_{ii} = 1$ and $q_{ii} = 1$

$\endgroup$
  • $\begingroup$ Thanks! I also think that in practice setting p = 0 doesn`t change anything. $\endgroup$ – Choung young jae Dec 5 '16 at 9:33
  • $\begingroup$ $p_{i|j}$ are supposed to sum to to 1 (when summed across $i$). How would this be possible or make sense if $p_{i|i}$ were equal to 1? This does not make sense to me. Zero is the only natural choice. $\endgroup$ – amoeba says Reinstate Monica Dec 18 '17 at 13:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.