I am creating a search engine, with the corpus consisting of websites crawled through a webcrawler (Apache Nutch). I need the query searches to be both fast and relevant. So far, I have been trying to develop a Latent Semantic Indexing system, but I recently read an article that told me that LSA had not shown to be useful to them. Basically, LSA uses a k-rank approximation of the singular value decomposition of a matrix whose dimensions are the documents and a weighted term frequency (IDF, TF-IDF, Binary, and Raw).

Does anyone know a much better data-mining algorithm that I could use? Through your own personal experience? I only have about 2 weeks left for this project, and I need to make sure I can end it right.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.