Which features to include for Truncated SVD? I have a dataset of ~31000 8k-filings (ad-hoc announcements from companies listed on the stock exchange). Every document consists of a string (the actual filing, stemmed and stopwords removed) and a few metadata-features i constructed, i.e. readability measures, number of words, number of adjectives per word and a list of organisations mentioned in the document. 
Along with market excess returns as dependent variable, I want to feed all the data as features to XGBoost.
For feature reduction, I first used TfidfVectorizers arguments min_df=5 and max_df to reduce bag-of-word features from 103000 down to 16000. 
Now I want to perform Truncated SVD for further feature reduction on this data. Scikitlearn's handbook suggests choosing k=100 for LSA. But I was asking myself if it would be better to only perform SVD only on the tfidf-bag-of-words and then add the metafeatures to the output or first add metafeatures and then perform SVD on all of the features? 
Is there a 'right' way to do this?
 A: Is this a multilabel classification problem? I am attempting to solve a similar problem (where the text response could be labeled with up to 6 possible target labels) with this approach:
1) Used TfidfVectorizers arguments (i.e. min_df=5 and max_df) to reduce bag-of-word features from a large number down to a smaller amount.
2) Perform TruncatedSVD with 100-300 components (still trying to figure out the best number of components).
3) Make pipeline of steps 1 and 2 representing your LSA results. From the Docs: When truncated SVD is applied to term-document matrices (as returned by CountVectorizer or TfidfVectorizer), this transformation is known as latent semantic analysis (LSA), because it transforms such matrices to a “semantic” space of low dimensionality.
4) Select classifier (i.e. Random Forest or other model that outputs probabilistic results) and LSA in final pipeline. Only use text features (actual filings) in this first model. Do not include structured/metadata features (i.e. readability measures, number of words, number of adjectives per word and a list of organizations). I am following this example for this step. 
5) Save predicted probabilities for each class label (if multilabel) as features for next model. 
6) Use these predicted probabilities with your metadata features in a final ensembled model. 
I will try and update this response (with code examples) when I have finished implementing a similar process and let you know how it goes. 
