You could use PCA or KernelPCA in a Pipeline to find out how many components suit your situation best. I am not too familiar with decomposition however, so take this with a grain of salt. KernelPCA + basic DecisionTree example: the estimator will create 3 folds for each `kpca__n_component` and compare their test results to find out what value performs better. You could also add parameters for your decision tree inside the pipeline using e.g. `dtree__min_samples_split=4`. from sklearn.decomposition import KernelPCA from sklearn.pipeline import Pipeline from sklearn.model_selection import GridSearchCV from sklearn.tree import DecisionTreeRegressor dtree = DecisionTreeRegressor() kpca = KernelPCA(kernel='rbf') pca_pipe = Pipeline(steps=[('kpca', kpca), ('dtree', dtree)]) estimator = GridSearchCV(pca_pipe, param_grid=dict( kpca__n_components = [4, 8, 12, 16, None])) estimator.fit(X, y) print estimator.best_params_, estimator.best_score_ The above code will print the best $R^2$ score and the corresponding n_components value of the KernelPCA object. This score is computed using DecisionTreeRegressor after transforming the data using KernelPCA. After the PCA transformation you can get the original features back using `kpca.inverse_transform([X_transformed])` so you can traceback what features were used in creating the tree. To truly improve your $R^2$ score I would suggest using an ensemble of trees such as GradientBoost or RandomForest instead of a single one, although this is not always feasible.