0
$\begingroup$

I am working with a data set that is 186x79. What I am interested in with this dataset is finding the features (predictors) that are the most important for predicting a response.

The actual data itself are different measurements of placentas at birth and the response is a log ratio of placenta weight and birth weight. I have my code set up like this in R.

placenta.model = lm(beta~., data=placenta)
summary(placenta.model)

Call:
lm(formula = beta ~ ., data = placenta)

Residuals:
      Min        1Q    Median        3Q       Max 
-0.054933 -0.014712  0.000189  0.012045  0.070035 

Coefficients: (5 not defined because of singularities)
                            Estimate Std. Error t value Pr(>|t|)   
(Intercept)                1.309e+00  1.385e+00   0.945  0.34656   
Area                      -1.445e-03  1.144e-03  -1.263  0.20920   
Perimeter                  1.934e-03  9.893e-03   0.195  0.84536   
Compactness               -7.428e-02  4.358e-01  -0.170  0.86498   
Eq_Circle_IP_Symmetry     -2.598e-02  9.705e-02  -0.268  0.78942   
Sigma_UCI                  2.492e-03  2.303e-02   0.108  0.91403   
Rmean                      7.697e-02  6.353e-02   1.211  0.22828   
RmeanN                    -9.282e-01  2.062e+00  -0.450  0.65354   
UCI_to_Perim              -2.906e-03  6.153e-03  -0.472  0.63765   
A_SurfaceArea              3.976e+01  5.637e+01   0.705  0.48203   
A_VesselToDiscPercent      7.125e+01  5.018e+01   1.420  0.15846   
A_NumGenerations           1.995e-03  1.580e-03   1.263  0.20922   
A_NumCordBranches         -4.416e-02  2.384e-02  -1.852  0.06667 . 
A_NumBranchPoints         -4.638e-02  2.417e-02  -1.919  0.05757 . 
A_NumEndPoints             4.590e-02  2.411e-02   1.904  0.05955 . 
A_ArcLength                1.029e+02  4.515e+01   2.279  0.02457 * 
A_Volume                  -1.794e+00  5.018e+01  -0.036  0.97155   
A_MeanThickness            9.540e-01  1.734e+00   0.550  0.58334   
A_StdThickness            -3.663e+00  2.606e+00  -1.405  0.16270   
A_MurrayBranchesUsed              NA         NA      NA       NA   
A_MurrayExponent          -1.445e-02  1.693e-02  -0.853  0.39547   
A_MurrayL1FitError         1.945e-01  7.519e-01   0.259  0.79633   
A_MeanDistToPerim          4.984e-02  2.252e-02   2.213  0.02898 * 
A_StdDistToPerim          -2.131e-02  2.954e-02  -0.722  0.47207   
A_MeanDistEndPointToPerim -1.727e-02  2.419e-02  -0.714  0.47682   
A_StdDistEndPointToPerim   9.405e-03  4.119e-02   0.228  0.81982   
A_MeanAngle               -1.034e-04  5.116e-04  -0.202  0.84020   
A_StdDevAngle              5.015e-05  1.121e-03   0.045  0.96441   
A_ModeAngle                2.200e-04  1.214e-04   1.812  0.07272 . 
A_MaxAngle                 1.246e-04  2.679e-04   0.465  0.64281   
A_MinAngle                -3.446e-04  3.906e-04  -0.882  0.37959   
A_KurtosisAngle           -6.367e-04  3.101e-03  -0.205  0.83770   
A_MeanTortuosity          -3.592e-01  3.762e-01  -0.955  0.34176   
A_StdDevTortuosity         5.496e-01  4.534e-01   1.212  0.22807   
A_MaxTortuosity           -5.506e-02  6.830e-02  -0.806  0.42190   
A_MinTortuosity           -3.022e-01  3.061e-01  -0.987  0.32564   
A_KurtosisTortuosity       2.760e-05  7.704e-04   0.036  0.97148   
V_SurfaceArea              3.976e+01  5.637e+01   0.705  0.48202   
V_VesselToDiscPercent      7.123e+01  5.018e+01   1.420  0.15854   
V_NumGenerations           6.395e-04  1.756e-03   0.364  0.71642   
V_NumCordBranches         -1.018e-02  2.993e-02  -0.340  0.73436   
V_NumBranchPoints         -1.038e-02  2.834e-02  -0.366  0.71474   
V_NumEndPoints             9.509e-03  2.826e-02   0.336  0.73715   
V_ArcLength                1.029e+02  4.515e+01   2.279  0.02457 * 
V_Volume                  -1.794e+00  5.018e+01  -0.036  0.97155   
V_MeanThickness            1.133e+00  1.537e+00   0.737  0.46241   
V_StdThickness            -3.052e+00  2.478e+00  -1.232  0.22070   
V_MurrayBranchesUsed              NA         NA      NA       NA   
V_MurrayExponent          -2.001e-03  1.130e-02  -0.177  0.85979   
V_MurrayL1FitError         1.749e-03  4.786e-01   0.004  0.99709   
V_MeanDistToPerim         -2.202e-02  1.961e-02  -1.123  0.26396   
V_StdDistToPerim           8.625e-02  2.753e-02   3.134  0.00221 **
V_MeanDistEndPointToPerim  2.807e-02  2.694e-02   1.042  0.29970   
V_StdDistEndPointToPerim  -3.163e-02  4.308e-02  -0.734  0.46434   
V_MeanAngle                8.149e-04  5.950e-04   1.369  0.17363   
V_StdDevAngle             -1.479e-03  1.082e-03  -1.366  0.17462   
V_ModeAngle               -1.447e-04  1.237e-04  -1.169  0.24483   
V_MaxAngle                 2.006e-04  2.661e-04   0.754  0.45259   
V_MinAngle                -7.271e-04  4.202e-04  -1.730  0.08632 . 
V_KurtosisAngle           -1.079e-03  2.452e-03  -0.440  0.66062   
V_MeanTortuosity          -1.324e-01  4.903e-01  -0.270  0.78758   
V_StdDevTortuosity        -6.027e-01  5.196e-01  -1.160  0.24860   
V_MaxTortuosity            1.013e-01  8.081e-02   1.254  0.21247   
V_MinTortuosity            1.763e-01  2.790e-01   0.632  0.52877   
V_KurtosisTortuosity      -1.019e-03  8.135e-04  -1.253  0.21301   
T_SurfaceArea             -3.976e+01  5.637e+01  -0.705  0.48202   
T_.VesselToDiscPercent    -7.123e+01  5.018e+01  -1.419  0.15856   
T_NumCordBranches                 NA         NA      NA       NA   
T_NumBranchPoints                 NA         NA      NA       NA   
T_NumEndPoints                    NA         NA      NA       NA   
T_ArcLength               -1.029e+02  4.515e+01  -2.279  0.02457 * 
T_Volume                   1.793e+00  5.018e+01   0.036  0.97156   
T_MeanThickness           -2.101e+00  3.147e+00  -0.668  0.50582   
T_StdThickness             6.105e+00  5.091e+00   1.199  0.23296   
T_MeanDistEndPointToPerim -2.551e-04  2.131e-02  -0.012  0.99047   
T_StdDistEndPointToPerim  -3.668e-04  4.054e-02  -0.009  0.99280   
T_MeanDistAEndToVEnd       4.997e-03  3.061e-02   0.163  0.87061   
T_StdDistAEndToVEnd        1.386e-02  2.543e-02   0.545  0.58686   
T_MeanDistAToV            -1.204e-01  9.042e-02  -1.332  0.18570   
T_StdDistAToV              1.276e-01  7.165e-02   1.781  0.07772 . 
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.0257 on 111 degrees of freedom
Multiple R-squared:  0.4178,    Adjusted R-squared:  0.02973 
F-statistic: 1.077 on 74 and 111 DF,  p-value: 0.3587

From what I understand my p-value is in favor of the null hypothesis that $\beta$=0 or similarly that a version of the model with less predictors performs better than the predictors I currently have. I aimed to solve this problem with glmnet to use feature importance. I tried to find the best parameters using caret only to receive the error:

Warning message:
In nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo,  :
  There were missing values in resampled performance measures.

From my understanding this is telling me that my $R^2$ statistic is constant over all my values. Please correct me if I am wrong. So from here I used the stepAIC model for feature selection, which actually gives me something tangible:

placenta.model = lm(beta~., data=placenta)
step <- stepAIC(placenta.model, direction="both")

Call:
lm(formula = beta ~ Area + Perimeter + Rmean + UCI_to_Perim + 
    A_VesselToDiscPercent + A_NumGenerations + A_NumCordBranches + 
    A_NumBranchPoints + A_NumEndPoints + A_ArcLength + A_StdThickness + 
    A_MurrayExponent + A_MeanDistToPerim + A_ModeAngle + A_MinAngle + 
    A_MinTortuosity + V_VesselToDiscPercent + V_ArcLength + V_StdDistToPerim + 
    V_MeanAngle + V_StdDevAngle + V_MinAngle + V_StdDevTortuosity + 
    T_.VesselToDiscPercent + T_ArcLength + T_StdDistAEndToVEnd + 
    T_MeanDistAToV + T_StdDistAToV + A_StdDistToPerim, data = placenta)

Residuals:
     Min       1Q   Median       3Q      Max 
-0.05454 -0.01598  0.00034  0.01570  0.07043 

Coefficients:
                         Estimate Std. Error t value Pr(>|t|)    
(Intercept)             7.582e-01  2.464e-01   3.077 0.002469 ** 
Area                   -1.401e-03  5.427e-04  -2.582 0.010740 *  
Perimeter               3.194e-03  1.480e-03   2.158 0.032450 *  
Rmean                   5.189e-02  2.967e-02   1.749 0.082289 .  
UCI_to_Perim           -2.042e-03  1.554e-03  -1.314 0.190696    
A_VesselToDiscPercent   6.090e+01  3.893e+01   1.564 0.119745    
A_NumGenerations        1.815e-03  1.089e-03   1.667 0.097578 .  
A_NumCordBranches      -3.796e-02  1.868e-02  -2.032 0.043838 *  
A_NumBranchPoints      -4.785e-02  1.880e-02  -2.545 0.011901 *  
A_NumEndPoints          4.670e-02  1.873e-02   2.493 0.013702 *  
A_ArcLength             1.142e+02  3.475e+01   3.286 0.001255 ** 
A_StdThickness         -6.210e-01  2.229e-01  -2.786 0.006006 ** 
A_MurrayExponent       -1.121e-02  5.619e-03  -1.995 0.047742 *  
A_MeanDistToPerim       3.333e-02  1.043e-02   3.194 0.001696 ** 
A_ModeAngle             1.513e-04  8.068e-05   1.875 0.062609 .  
A_MinAngle             -4.074e-04  2.797e-04  -1.457 0.147221    
A_MinTortuosity        -5.073e-01  2.189e-01  -2.317 0.021807 *  
V_VesselToDiscPercent   6.090e+01  3.893e+01   1.564 0.119788    
V_ArcLength             1.142e+02  3.475e+01   3.286 0.001256 ** 
V_StdDistToPerim        6.415e-02  1.904e-02   3.369 0.000951 ***
V_MeanAngle             7.969e-04  3.466e-04   2.299 0.022832 *  
V_StdDevAngle          -3.694e-04  2.835e-04  -1.303 0.194500    
V_MinAngle             -4.614e-04  3.186e-04  -1.448 0.149529    
V_StdDevTortuosity     -1.032e-01  7.406e-02  -1.394 0.165445    
T_.VesselToDiscPercent -6.089e+01  3.893e+01  -1.564 0.119798    
T_ArcLength            -1.142e+02  3.475e+01  -3.286 0.001256 ** 
T_StdDistAEndToVEnd     2.207e-02  1.285e-02   1.717 0.087944 .  
T_MeanDistAToV         -1.039e-01  6.254e-02  -1.661 0.098640 .  
T_StdDistAToV           1.222e-01  5.427e-02   2.252 0.025703 *  
A_StdDistToPerim       -3.116e-02  2.171e-02  -1.436 0.153122    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.02309 on 156 degrees of freedom
Multiple R-squared:  0.3394,    Adjusted R-squared:  0.2167 
F-statistic: 2.764 on 29 and 156 DF,  p-value: 3.023e-05

This seems great but I have read in literature that this is not a good way to proceed. So my question here is:

  1. When I am trying to fit a linear model and my predictors provider a high p-value what is the best way to proceed? I thought a good way to proceed were by some of the solutions posted above.

  2. What do the NAs represent in the first model?

Anything else that anyone can suggest would be great! Thank you!

$\endgroup$
  • $\begingroup$ When you say your "R^2 statistic is constant over all the values." What values are you referring to? Predictive accuracy after fitting models with any subset of model predictors? $\endgroup$ – AdamO Nov 2 '15 at 18:54
  • $\begingroup$ @AdamO - check this out - stackoverflow.com/questions/10503784/… $\endgroup$ – RDizzl3 Nov 2 '15 at 18:59
  • 1
    $\begingroup$ I think I see. I suspect the singularity of the design matrix is largely at play. I would encourage you to systematically evaluate your predictors and throw out the ones you know to be bunk. That's the ideal first place to start. $\endgroup$ – AdamO Nov 2 '15 at 19:08
  • $\begingroup$ @AdamO - that is a good idea. I was also thinking along the lines of feature engineering or interactions between variables. To try to see if I can actually engineer something with a little more predictive power. Of course this would require research on this area to begin with. Good think I'm paired with a biology expert :) $\endgroup$ – RDizzl3 Nov 2 '15 at 19:10
  • 1
    $\begingroup$ I agree strongly with @AdamO. For example, I'd expect Area and Perimeter to be very highly correlated. I'd advise a thorough exploratory data analysis (correlation matrix, scatter plot matrix, consideration of transformations, etc.). While not underplaying the role of domain knowledge, many of the ideas concerned feature in any good first statistics course. $\endgroup$ – Nick Cox Nov 2 '15 at 19:11
1
$\begingroup$

Since you are working with nested models, a much better approach might be done with plain LASSO. From what I understand, GLMNET concerns estimating a non-linear relationship between several factors and the outcome of interest, which is generally important if there is one or more factors having a complex non-linear relationship with the outcome. With less than 200 observations and nearly half as many features, the idea of estimating this reliably is ludicrous.

The missing coefficients in your kitchen sink model is a testament to that. The NA values are a consequence of singularity, when one column or more of the model matrix can be expressed as a linear combination of other columns. Therefore if stepwise model selection is the appropriate means of creating a final predictive model, forward stepwise selection is the only viable solution: and iffy at that.

LASSO could be much better. I think you should give that an attempt if this really concerns building a predictive model.

Nonetheless, I can't help but observe that these measurements seem to arise from a somewhat sophisticated physical process. Building predictive models has limited importance in fields where processes are not understood. However, if a process is understood well enough to formulate a model (or two or ten) without relying on complicated machine learning techniques, I wholeheartedly advocate abandoning ML methods in favor of --what I call-- fitting a "physical" model. This is much simpler and easier, usually. Just see the R help entry for nls. Physical model(s) generally provides a fantastic means of communicating data to physicists, biologists, or other groups. Finding 4 or 5 physical models, and comparing AIC provides some means of comparing their predictive accuracy.

$\endgroup$
  • $\begingroup$ I will try that out! You are correct about the measurements as well. This data is highly specialized to biology centered around child birth and the actual goal itself is to see which measurements are the most important in determining the response I mentioned. Thank you for your response! $\endgroup$ – RDizzl3 Nov 2 '15 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.