Using latent Dirichlet allocation for information retrieval I am working on understanding various document ranking algorithms like (TF-IDF, LSI, language models, etc) by actually implementing them. I want to understand LDA and using various resources to understand the algorithm. What I don't understand is how we come up with the latent (hidden) variables/topics. Can someone please explain it to me using examples like:
Doc1: "shipment of gold damaged in a fire",
Doc2: "delivery of silver arrived in a silver truck", 
Doc3: "shipment of gold arrived in a truck"}; 
Query: "gold silver truck"

I will really appreciate any help in this regard. Thanks! 
 A: At a very high-level view, latent topics are formed from words that often appear together in the same documents.
Your examples don't have a clear set of topics, so let's use the following documents instead:
Doc1: After I eat my breakfast of apples, oranges, bananas, and grapes, I'm going to go snowboarding in the Alps if it's not too cold outside.
Doc2: Apples, oranges, bananas, and grapes make good smoothies.
Doc3: Apples, oranges, bananas, and grapes are tasty fruits.
Doc4: Snowboarding in the Alps is a lot of fun, but cold.
Doc5: My boyfriend lives in the Alps, where he teaches snowboarding.

Suppose we say there are two latent topics that we want to discover. The topics that we discover are likely to be:


*

*Topic 1 (the "fruit" topic): represented most strongly by apples, oranges, bananas, grapes.

*Topic 2 (the "Alps" topic): represented most strongly by Alps, snowboarding, cold.


Doc 1 is then about an equal mix of topic 1 and topic 2, docs 2-3 are mostly topic 1, docs 4-5 are mostly topic 2.
Here's an interesting example of latent dirichlet allocation applied to the WikiLeaks CableGate: http://idea.ed.ac.uk/topics/cables/browser/cables.html (The set of topics are on the left.)
Also, I wasn't sure if you wanted a high-level view or a more technical algorithmic explanation, so if it's the latter you were looking for, just say so and I can add a more technical explanation.
