I've been also trying to use Neural Networks for text categorization/classification with limited success. I tried to move away from unigram/bigram features (very sparse, very high-dimensional) to dense and much smaller dimensionality representations. I tried LDA (Latent Dirichlet Allocation) and some other feature selection/extraction methods but only got inferior performance compared to sparse unigram/bigram features used in Logistic Regression.
I am well aware of recent papers using Recurrent Neural Networks and other deep learning techniques but they need lots of data and demand significant computing power. While the latter I have, in my application I don't possess lots of data. So I must stick with shallow machine learning methods.
I am very interested to find out what dense and low-dimensional features give at least comparable performance to unigrams/bigrams in a setting when datasets are not large enough for deep learning. I am particularly interested in methods analyzing/mining short text documents.