The stochastic part is that you will pick data points at random. Unless you only have one [$x_1, x_2, x_3$] vector, then you'll just 'stochastically' pick that one over and over again. Have never heard of stochastically updating weights (although it could be useful it would probably just be inefficient) - all weights are always updated at every training step.

I mean, that's a pretty difficult one. You could perhaps give a topic the most frequent word in documents of that topic? Or maybe better still, give it the most 'discriminative' word, which you could define as perhaps the word with the highest (occurrence rate in that topic's documents) / (average occurrence rate over all documents)

Edit: works better if you normalise the latent vectors pastebin.com/VRLRjTFL

No worries. And I think you might still be able to take the approach I suggested. I created a toy example with Tensorflow here pastebin.com/s9TWB68p . Basically I made synthetic pairing probabilities like the data you have, and then learned an arbitrary latent space based on just these pairings, which ends up separating the 'true' classes fairly well. I don't know if the approach is that robust but it might be worth a try - let me know if it helps!

Ah yep good point. Didn't look at the link in enough detail, sorry. There must be approaches out there for optimising the cliques as you say I would have thought. A quick google turned up this sciencedirect.com/science/article/pii/S1877050914004256 ? Another thought: maybe you could do something similar to the siamese network but without neural nets. Just have a latent vector z for each sentence, then perform stochastic gradient descent on these z's where the cost is cross-entropy between their cosine similarity and the target probability. Might work?

I think I'm a bit clearer on what you mean now. If the words are really completely meaningless, then maybe you could look for some sort of graph-based clustering method? Nodes are sentences, edges would be the pairing probabilities, and you can treat unknown edges as latent. Maybe even this? tkipf.github.io/graph-convolutional-networks

Well if your sentences are meaningless then you can't really expect to extract any representation from them I suppose... Isn't the N-hot approach still feasible here? Edit:

Not really. This won't make the network any more (or less) robust as per dropout. It's closer in principle to subsampling an over-represented class when dealing with class-imbalanced data.