With my 37 base features, I obtain a PCA 7 components whose explained variance is 62584.661. Below is my code (I use scikit-learn):

>> pca = decomposition.PCA()
>> X = set_1.to_dataframe().drop(['user_id', 'TARGET'], axis=1)
>> pca.fit(X)

>> print pca.explained_variance_

  [6.14280876e+04   4.54217662e+01   2.72989436e+01   2.67631322e+00
   1.82282196e+00   1.21712136e+00   1.14632304e+00   9.78220983e-01
   5.29859226e-01   4.46864198e-01   2.20896621e-01   1.35477040e-01
   1.24036813e-01   1.16983213e-01   9.77577393e-02   9.52879856e-02
   6.33013163e-02   5.11503915e-02   4.46737594e-02   4.17721621e-02
   3.62836939e-02   2.81101420e-02   2.40227513e-02   1.73247899e-02
   9.59248897e-03   5.97952400e-03   2.32278409e-03   1.53377399e-03
   1.13924785e-03   3.69972626e-04   5.94762833e-28   5.94762833e-28
   5.94762833e-28   5.94762833e-28   5.94762833e-28]

 >> pca.explained_variance_[:7].sum()
 62584.661944243671

Using these 7 PCA variables in my linear regression model, I obtain 0.20 as the r2 value.

To improve the results, I add 39 more features (76 in total). Following the same steps, PCA results indicate 71052.645592029236 as the variance explained with 76 components. However, when I run linear regression using these 76 components, I obtain a r2 of 0.06 that is less than 0.20 of the previous model.

I wonder if I am doing something wrong in my experiment. Is this behavior expected? I thought I would get a better model with a PCA that explains more variance.

With my 37 base features, I obtain a PCA 7 components whose explained variance is 62584.661. Below is my code (I use scikit-learn):

>> pca = decomposition.PCA()
>> X = set_1.to_dataframe().drop(['user_id', 'TARGET'], axis=1)
>> pca.fit(X)

>> print pca.explained_variance_

  [6.14280876e+04   4.54217662e+01   2.72989436e+01   2.67631322e+00
   1.82282196e+00   1.21712136e+00   1.14632304e+00   9.78220983e-01
   5.29859226e-01   4.46864198e-01   2.20896621e-01   1.35477040e-01
   1.24036813e-01   1.16983213e-01   9.77577393e-02   9.52879856e-02
   6.33013163e-02   5.11503915e-02   4.46737594e-02   4.17721621e-02
   3.62836939e-02   2.81101420e-02   2.40227513e-02   1.73247899e-02
   9.59248897e-03   5.97952400e-03   2.32278409e-03   1.53377399e-03
   1.13924785e-03   3.69972626e-04   5.94762833e-28   5.94762833e-28
   5.94762833e-28   5.94762833e-28   5.94762833e-28]

 >> pca.explained_variance_[:7].sum()
 62584.661944243671

Using these 7 PCA variables in my linear regression model, I obtain 0.20 as the r2 value.

To improve the results, I add 39 more features (76 in total). Following the same steps, PCA results indicate 71052.64 as the variance explained with 76 components. However, when I run linear regression using these 76 components, I obtain a r2 of 0.06 that is less than 0.20 of the previous model.

I wonder if I am doing something wrong in my experiment. Is this behavior expected? I thought I would get a better model with a PCA that explains more variance.