I'm playing around with random forest classification and principal component analysis using scikit-learn and have found a point of confusion. I want to fit two models that predict 1 of 5 target classifications - (1) a random forest classification using all 53 features in my data, (2) a random forest classification using 2 principal components after reducing the data using PCA. I know that this is not really necessary with random forests, but I'm more just doing this for fun and to see how the results compare.

My random forest classification using all features has a very high out of sample accuracy (99%), but my reduced model using PCA has a very low out of sample accuracy (20%). Given that there are only 5 classifications, this is the accuracy we would expect if we randomly classified. So - something went wrong. I imagine I just made an error in my code, but I can't seem to figure it out. Is there anything obvious in my code below that I messed up?

FYI: I'm using the human activity recognition weight lifting dataset found here: http://groupware.les.inf.puc-rio.br/har. The dataset contains various measurements of human movement and has a "classification" for the type of exercise done. I am trying to build a random forest model that predicts the exercise class.

>>> import pandas as pd
>>> import numpy as np
>>> from sklearn.cross_validation import train_test_split
>>> from sklearn.ensemble import RandomForestClassifier 
>>> from sklearn.metrics import *
>>> import matplotlib.pyplot as plt
>>> from sklearn.feature_selection import VarianceThreshold
>>> from sklearn.decomposition import PCA
>>> from sklearn.metrics import accuracy_score

First read in the raw data and get rid of certain columns. This includes the first 7 columns which have irrelevant ID info, and columns that contains missing data. Additionally I get rid of the "classe" variable which is what I want to predict. I then normalize the data and split it into training and test sets.

>>> raw_data=pd.read_csv('pml-training.csv', na_values=["NA","#DIV/0!"])
>>> data=raw_data[raw_data.columns[7:]].dropna(axis=1, how='any').drop('classe',1).values
>>> target=raw_data.classe.values
>>> data=(data-data.mean(axis=0))/data.std(axis=0)
>>> xtrain, xtest, ytrain, ytest =train_test_split(data, target, test_size=0.3, random_state=519)

I first use a random forest model and fit it on my training data, predict it on my test data, and calculate the out of sample accuracy, which is 99%.

>>> forest_mod = RandomForestClassifier(random_state=118, n_estimators = 50)
>>> forest=forest_mod.fit(xtrain,ytrain)
>>> y_pred_forest=forest.predict(xtest)
>>> print accuracy_score(ytest, y_pred_forest)

Then I want to try to reduce the data to two principal components and use those two components to build a random forest model and compare the accuracy. First I create the model with 2 components and use it to fit and transform both the training and test datasets.

>>> pca_mod = PCA(2)
>>> xtrain_pca = pca_mod.fit_transform(xtrain)
>>> xtest_pca=pca_mod.fit_transform(xtest)

Then I use the random forest to build a prediction using the newly transformed x training data. And just like above, I use it on my test data to calculate the out of sample error rate.

>>> forest_pca=forest_mod.fit(xtrain_pca,ytrain)
>>> y_pred_pca=forest_pca.predict(xtest_pca)
>>> print accuracy_score(ytest, y_pred_pca)


I don't think this is a duplicate because as shown in the comments PCA is not causing my results to worsen because of any statistical reason, but there is instead an error in how it is being applied. Even when setting the number of PCAs equal to the number of features, it only predicted 25% on my test data, which is not realistic given a regular random forest predicted 99%. Even using just 2 PCAs explains 34% of the variance.

I think that PCA is being executed correctly but there must be an issue in how the results of the PCA are being applied to the test dataset in RF (last 2 blocks of code above).

I'm not sure if this will help diagnosis the issue, but I plotted the 2 PCAs that were run on the training set and colored them by which of the 5 groups they belong to.

>>> colors = {'A':'red', 'B':'blue', 'C':'green', 'D':'black','E':'yellow'}
>>> y_train_colors= np.copy(ytrain)
>>> for k, v in colors.iteritems(): y_train_colors[ytrain==k] = v
>>> plt.scatter(x=xtrain_pca[:,0], y=xtrain_pca[:,1], c=y_train_colors.tolist())

enter image description here

5 groups are very clearly being defined by these two PCAs but the colors don't match as I would expect. Any thoughts on where this went awry?

EDIT 2 I've decided to try and simplify the problem to see why my PCA does not appear to be giving me the results I expect to see. I run the same set of code on the sample Iris dataset that comes with sklearn as well as on the human activity recognition dataset. The results seem make sense when applied to the Iris dataset, but not when applied the human activity dataset. This indicates to me that perhaps the issue is not with my code, but some misunderstanding of the data.

I use the following code to reduce the Iris data to two principal components and then plot them colored by target group. As you can see the PCA seems to have done a good job identifying the 3 different groups.

from sklearn import datasets
from sklearn.decomposition import PCA
import pandas as pd

#load Iris data
iris = datasets.load_iris()
X = iris.data
Y = iris.target

#standardize X Values
X_standard= (X-X.mean(axis = 0))/X.std(axis = 0)

#Reduced to two components and get new X values
pca = PCA(2)
X_fit = pca.fit_transform(X_standard)

#build a dataframe with principal components and with target group
df = pd.DataFrame(X_fit, columns = ['PC1','PC2'])

#map target to specified colors
colors=  df['Group'].map({0:'red', 1:'blue', 2:'yellow'})

df.plot.scatter('PC1', 'PC2', c=colors)

enter image description here

I then run the same code on the human activity recognition dataset. Note that I'm running the this on a slightly different version than I did initially, which is why the scatter plot looks a little different than the one above (also the principal components are plotted in the reverse order).

Nonetheless, as you can see, the PCA does a pretty good job dividing the dataset into 5 target groups (of which there are), but the colors that identify the groups do not indicate that the data is being partitioned correctly. Is this possible?


#don't use first 6 columns and drop columns with nulls

X=raw_data[raw_data.columns[6:]].dropna(axis=1, how='any').drop('classe',1) 

#standardize X Values
X_standard= (X-X.mean(axis = 0))/X.std(axis = 0)

#Reduced to two components and get new X values
pca = PCA(2)
X_fit = pca.fit_transform(X_standard)

#build a dataframe with principal components and with target group
df = pd.DataFrame(X_fit, columns = ['PC1','PC2'])

#map target to specified colors
colors = df['Group'].map({'A':'red', 'B':'blue', 'C':'green', 'D':'black','E':'yellow'})

df.plot.scatter('PC1', 'PC2', c=colors)

enter image description here

  • $\begingroup$ To check for coding mistakes, run the same thing with more and more PCs included. With large number of PCs you should be getting close to 99% accuracy. Other than that, this seems to be a duplicate so I am voting to close. $\endgroup$
    – amoeba
    Jul 30 '16 at 23:20
  • $\begingroup$ See also my answer here for more links: stats.stackexchange.com/questions/141864. $\endgroup$
    – amoeba
    Jul 30 '16 at 23:21
  • 1
    $\begingroup$ @amoeba good question - when I include 52 PCs (equal to the number of original features) the accuracy is still only 25.1%. This indicates to me that something must be wrong with the code. $\endgroup$
    – AJG519
    Jul 31 '16 at 5:20
  • 1
    $\begingroup$ I agree that it's not a duplicate anymore, but now it's a request in the form "Here is my code, please help me find a bug", which are arguably off-topic here :-( I will take another look at your code though. $\endgroup$
    – amoeba
    Aug 5 '16 at 21:15
  • 2
    $\begingroup$ I am not sure if this is the cause of your problem, but I found one conceptual error. You apply PCA to train and to test separately (i.e. you run PCA on train and PCA on test). This is wrong. You should apply PCA to train, and then use the same linear transformation on the test data. $\endgroup$
    – amoeba
    Aug 5 '16 at 22:24