Is it a good idea to use a linear model (like logistic regression) to generate new features for a non linear model (like random forest)? The setting is a 2-class classification problem. We have too many features, some of them not very informative and with many zeros. We are thinking in ways of selecting the best features, and PCA (in the full dataset or maybe in groups of related features) is one of the alternatives. But I thought if there was another way of generating linear combinations of features that not only takes in consideration the intrinsic variance, but also the relationship with the target. Like a target-PCA, if that existed.
And an approximation of this idea could be what I ask in the main question: Could it be a good idea to use a linear classifier like logistic regression or SVM to generate linear combinations of features that, in a way, are optimizing the information gain with respect to the target? Because I think that if, given a subset of variables, a hyperplane can give a good separation of the classes, the equation of the hyperplane, considered as a feature, have more predictive power than any of the individual features, so maybe you could substitute the group of features with the new one and give all this generated features to the last model (the random forest) as the inputs.
EDIT: There is a very similar question to this one, that someone has suggested:
Non-perpendicular hyperplane decision trees
It's closely related to what I was thinking. Thanks everyone!!
 A: *

*It looks like Partial Least Squares (PLS) is what you call "target-PCA" - originally this is for regression, but there are versions for classification.


*One problem with what you propose here is that you will need to be careful when later using something like cross-validation for assessing the quality of your classifier, because if you use the whole dataset for feature generation, cross-validation of the later random forest will be misleading. (This can be dealt with cross-validating the whole process, but that's more difficult and computationally more cumbersome.)


*I'd be surprised if information reduction before random forest is better than random forest on the full information - I don't know of any results that would suggest that such an operation in advance somehow helps the random forest, although the possibility that it does in your situation cannot be excluded (if you have enough data you could leave some aside and compare).


*Another issue is that logistic regression or SVD generate features in order to optimise their own way of classifying - why should it be better to use these features with another method that was set up to do something else?
