You could use PCA or KernelPCA in a Pipeline to find out how many components suit your situation best. I am not too familiar with decomposition however, so take this with a grain of salt. KernelPCA + basic DecisionTree example: the estimator will create 3 folds for each kpca__n_component and compare their test results to find out what value performs better. You could also add parameters for your decision tree inside the pipeline using e.g. dtree__min_samples_split=4. from sklearn.decomposition import KernelPCA from sklearn.pipeline import Pipeline from sklearn.model_selection import GridSearchCV from sklearn.tree import DecisionTreeRegressor dtree = DecisionTreeRegressor() kpca = KernelPCA(kernel='rbf') pca_pipe = Pipeline(steps=[('kpca', kpca), ('dtree', dtree)]) estimator = GridSearchCV(pca_pipe, param_grid=dict( kpca__n_components = [4, 8, 12, 16, None])) estimator.fit(X, y) print estimator.best_params_, estimator.best_score_ The above code will print the best R^2 score and the corresponding n_components value of the KernelPCA object. This score is computed using DecisionTreeRegressor after transforming the data using KernelPCA. After the PCA transformation you can get the original features back using kpca.inverse_transform([X_transformed]) so you can traceback what features were used in creating the tree. To truly improve your R^2 score I would suggest using an ensemble of trees such as GradientBoost or RandomForest instead of a single one, although this is not always feasible.