How should I use PCA to identify features for Regression tree analysis?

Question

I have a data set of 932 responses (carbon emissions) with p=21 features (Temperature, net radiation, photon flux density, soil water content etc.) that may have some influence on them.

I am using scikit learn's DecisionTreeRegressor and the Docs suggest using PCA to help narrow down the number of features to look at. Some of my factors are definitely highly correlated eg. net radiation and photon flux so using PCA sounds like a good idea.

I understand the general principal of PCA, but I'm having a hard time finding any information of how I should use PCA to do this. Should I use the first n principal components as the input features for the regression tree? Here is a plot of the explained variance ratio for my dataset generated using scikit's PCA:

Once of the pro's of CART is that trees are easy to interpret, but I don't see how that result would be easy to explain or be applicable to any other data sets? Or is there some standard way to use the principal components to identify the most "important" factors? I've seen mention of using the correlation between factors and the first n principal components but I can't seem to find anything on it now. Anything to help point me in the right direction would be greatly appreciated!

Answer 1

You could use PCA or KernelPCA in a Pipeline to find out how many components suit your situation best. I am not too familiar with decomposition however, so take this with a grain of salt.

KernelPCA + basic DecisionTree example: the estimator will create 3 folds for each kpca__n_component and compare their test results to find out what value performs better. You could also add parameters for your decision tree inside the pipeline using e.g. dtree__min_samples_split=4.

from sklearn.decomposition import KernelPCA
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.tree import DecisionTreeRegressor

dtree = DecisionTreeRegressor()
kpca = KernelPCA(kernel='rbf')

pca_pipe = Pipeline(steps=[('kpca', kpca), ('dtree', dtree)])

estimator = GridSearchCV(pca_pipe, param_grid=dict(
                    kpca__n_components = [4, 8, 12, 16, None]))
estimator.fit(X, y)

print estimator.best_params_, estimator.best_score_

The above code will print the best $R^2$ score and the corresponding n_components value of the KernelPCA object. This score is computed using DecisionTreeRegressor after transforming the data using KernelPCA. After the PCA transformation you can get the original features back using kpca.inverse_transform([X_transformed]) so you can traceback what features were used in creating the tree.

To truly improve your $R^2$ score I would suggest using an ensemble of trees such as GradientBoost or RandomForest instead of a single one, although this is not always feasible.