1
$\begingroup$

With my 37 base features, I obtain a PCA 7 components whose explained variance is 62584.661. Below is my code (I use scikit-learn):

>> pca = decomposition.PCA()
>> X = set_1.to_dataframe().drop(['user_id', 'TARGET'], axis=1)
>> pca.fit(X)

>> print pca.explained_variance_

  [6.14280876e+04   4.54217662e+01   2.72989436e+01   2.67631322e+00
   1.82282196e+00   1.21712136e+00   1.14632304e+00   9.78220983e-01
   5.29859226e-01   4.46864198e-01   2.20896621e-01   1.35477040e-01
   1.24036813e-01   1.16983213e-01   9.77577393e-02   9.52879856e-02
   6.33013163e-02   5.11503915e-02   4.46737594e-02   4.17721621e-02
   3.62836939e-02   2.81101420e-02   2.40227513e-02   1.73247899e-02
   9.59248897e-03   5.97952400e-03   2.32278409e-03   1.53377399e-03
   1.13924785e-03   3.69972626e-04   5.94762833e-28   5.94762833e-28
   5.94762833e-28   5.94762833e-28   5.94762833e-28]

 >> pca.explained_variance_[:7].sum()
 62584.661944243671

Using these 7 PCA variables in my linear regression model, I obtain 0.20 as the r2 value.

To improve the results, I add 39 more features (76 in total). Following the same steps, PCA results indicate 71052.64 as the variance explained with 76 components. However, when I run linear regression using these 76 components, I obtain a r2 of 0.06 that is less than 0.20 of the previous model.

I wonder if I am doing something wrong in my experiment. Is this behavior expected? I thought I would get a better model with a PCA that explains more variance.

Improve this question
$\endgroup$
15
  • $\begingroup$ These r-squared values are from the training set or from the test set? Or obtained via cross-validation? $\endgroup$
    – amoeba
    Commented Feb 14, 2017 at 10:02
  • 1
    $\begingroup$ @amoeba From the test set. I split the data into 2 partition: 0.7 (training), and 0.3 (testing). You think this behaviour is weird? $\endgroup$
    – renakre
    Commented Feb 14, 2017 at 10:04
  • $\begingroup$ I see. No, it's not weird. I think your question is a duplicate, wait a second - I will provide a link. $\endgroup$
    – amoeba
    Commented Feb 14, 2017 at 10:05
  • $\begingroup$ @amoeba thanks for the link! It helps but I still do not know what should be my next move? $\endgroup$
    – renakre
    Commented Feb 14, 2017 at 10:08
  • $\begingroup$ What do you mean? Next move in order to achieve what? $\endgroup$
    – amoeba
    Commented Feb 14, 2017 at 10:09

0

Reset to default