Lucene-based text feature construction

Question

When doing the feature construction for text mining, does Lucene has a better performance in terms of classification/clustering result than the traditional bag-of-word approach?

Answer 1

Lucene analyzers gives you more flexibility that most naive feature extractors (language specific stemming & lemmatization, ICU normalization, n-grams and shingle based tokenization...).

It's data / problem / user dependent to determine if that flexibility will allow you to build better models or just add noisy feature in your preprocessed dataset.