# How to proceed when a linear model has a high p-value?

I am working with a data set that is 186x79. What I am interested in with this dataset is finding the features (predictors) that are the most important for predicting a response.

The actual data itself are different measurements of placentas at birth and the response is a log ratio of placenta weight and birth weight. I have my code set up like this in R.

placenta.model = lm(beta~., data=placenta)
summary(placenta.model)

Call:
lm(formula = beta ~ ., data = placenta)

Residuals:
Min        1Q    Median        3Q       Max
-0.054933 -0.014712  0.000189  0.012045  0.070035

Coefficients: (5 not defined because of singularities)
Estimate Std. Error t value Pr(>|t|)
(Intercept)                1.309e+00  1.385e+00   0.945  0.34656
Area                      -1.445e-03  1.144e-03  -1.263  0.20920
Perimeter                  1.934e-03  9.893e-03   0.195  0.84536
Compactness               -7.428e-02  4.358e-01  -0.170  0.86498
Eq_Circle_IP_Symmetry     -2.598e-02  9.705e-02  -0.268  0.78942
Sigma_UCI                  2.492e-03  2.303e-02   0.108  0.91403
Rmean                      7.697e-02  6.353e-02   1.211  0.22828
RmeanN                    -9.282e-01  2.062e+00  -0.450  0.65354
UCI_to_Perim              -2.906e-03  6.153e-03  -0.472  0.63765
A_SurfaceArea              3.976e+01  5.637e+01   0.705  0.48203
A_VesselToDiscPercent      7.125e+01  5.018e+01   1.420  0.15846
A_NumGenerations           1.995e-03  1.580e-03   1.263  0.20922
A_NumCordBranches         -4.416e-02  2.384e-02  -1.852  0.06667 .
A_NumBranchPoints         -4.638e-02  2.417e-02  -1.919  0.05757 .
A_NumEndPoints             4.590e-02  2.411e-02   1.904  0.05955 .
A_ArcLength                1.029e+02  4.515e+01   2.279  0.02457 *
A_Volume                  -1.794e+00  5.018e+01  -0.036  0.97155
A_MeanThickness            9.540e-01  1.734e+00   0.550  0.58334
A_StdThickness            -3.663e+00  2.606e+00  -1.405  0.16270
A_MurrayBranchesUsed              NA         NA      NA       NA
A_MurrayExponent          -1.445e-02  1.693e-02  -0.853  0.39547
A_MurrayL1FitError         1.945e-01  7.519e-01   0.259  0.79633
A_MeanDistToPerim          4.984e-02  2.252e-02   2.213  0.02898 *
A_StdDistToPerim          -2.131e-02  2.954e-02  -0.722  0.47207
A_MeanDistEndPointToPerim -1.727e-02  2.419e-02  -0.714  0.47682
A_StdDistEndPointToPerim   9.405e-03  4.119e-02   0.228  0.81982
A_MeanAngle               -1.034e-04  5.116e-04  -0.202  0.84020
A_StdDevAngle              5.015e-05  1.121e-03   0.045  0.96441
A_ModeAngle                2.200e-04  1.214e-04   1.812  0.07272 .
A_MaxAngle                 1.246e-04  2.679e-04   0.465  0.64281
A_MinAngle                -3.446e-04  3.906e-04  -0.882  0.37959
A_KurtosisAngle           -6.367e-04  3.101e-03  -0.205  0.83770
A_MeanTortuosity          -3.592e-01  3.762e-01  -0.955  0.34176
A_StdDevTortuosity         5.496e-01  4.534e-01   1.212  0.22807
A_MaxTortuosity           -5.506e-02  6.830e-02  -0.806  0.42190
A_MinTortuosity           -3.022e-01  3.061e-01  -0.987  0.32564
A_KurtosisTortuosity       2.760e-05  7.704e-04   0.036  0.97148
V_SurfaceArea              3.976e+01  5.637e+01   0.705  0.48202
V_VesselToDiscPercent      7.123e+01  5.018e+01   1.420  0.15854
V_NumGenerations           6.395e-04  1.756e-03   0.364  0.71642
V_NumCordBranches         -1.018e-02  2.993e-02  -0.340  0.73436
V_NumBranchPoints         -1.038e-02  2.834e-02  -0.366  0.71474
V_NumEndPoints             9.509e-03  2.826e-02   0.336  0.73715
V_ArcLength                1.029e+02  4.515e+01   2.279  0.02457 *
V_Volume                  -1.794e+00  5.018e+01  -0.036  0.97155
V_MeanThickness            1.133e+00  1.537e+00   0.737  0.46241
V_StdThickness            -3.052e+00  2.478e+00  -1.232  0.22070
V_MurrayBranchesUsed              NA         NA      NA       NA
V_MurrayExponent          -2.001e-03  1.130e-02  -0.177  0.85979
V_MurrayL1FitError         1.749e-03  4.786e-01   0.004  0.99709
V_MeanDistToPerim         -2.202e-02  1.961e-02  -1.123  0.26396
V_StdDistToPerim           8.625e-02  2.753e-02   3.134  0.00221 **
V_MeanDistEndPointToPerim  2.807e-02  2.694e-02   1.042  0.29970
V_StdDistEndPointToPerim  -3.163e-02  4.308e-02  -0.734  0.46434
V_MeanAngle                8.149e-04  5.950e-04   1.369  0.17363
V_StdDevAngle             -1.479e-03  1.082e-03  -1.366  0.17462
V_ModeAngle               -1.447e-04  1.237e-04  -1.169  0.24483
V_MaxAngle                 2.006e-04  2.661e-04   0.754  0.45259
V_MinAngle                -7.271e-04  4.202e-04  -1.730  0.08632 .
V_KurtosisAngle           -1.079e-03  2.452e-03  -0.440  0.66062
V_MeanTortuosity          -1.324e-01  4.903e-01  -0.270  0.78758
V_StdDevTortuosity        -6.027e-01  5.196e-01  -1.160  0.24860
V_MaxTortuosity            1.013e-01  8.081e-02   1.254  0.21247
V_MinTortuosity            1.763e-01  2.790e-01   0.632  0.52877
V_KurtosisTortuosity      -1.019e-03  8.135e-04  -1.253  0.21301
T_SurfaceArea             -3.976e+01  5.637e+01  -0.705  0.48202
T_.VesselToDiscPercent    -7.123e+01  5.018e+01  -1.419  0.15856
T_NumCordBranches                 NA         NA      NA       NA
T_NumBranchPoints                 NA         NA      NA       NA
T_NumEndPoints                    NA         NA      NA       NA
T_ArcLength               -1.029e+02  4.515e+01  -2.279  0.02457 *
T_Volume                   1.793e+00  5.018e+01   0.036  0.97156
T_MeanThickness           -2.101e+00  3.147e+00  -0.668  0.50582
T_StdThickness             6.105e+00  5.091e+00   1.199  0.23296
T_MeanDistEndPointToPerim -2.551e-04  2.131e-02  -0.012  0.99047
T_StdDistEndPointToPerim  -3.668e-04  4.054e-02  -0.009  0.99280
T_MeanDistAEndToVEnd       4.997e-03  3.061e-02   0.163  0.87061
T_StdDistAEndToVEnd        1.386e-02  2.543e-02   0.545  0.58686
T_MeanDistAToV            -1.204e-01  9.042e-02  -1.332  0.18570
T_StdDistAToV              1.276e-01  7.165e-02   1.781  0.07772 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.0257 on 111 degrees of freedom
Multiple R-squared:  0.4178,    Adjusted R-squared:  0.02973
F-statistic: 1.077 on 74 and 111 DF,  p-value: 0.3587


From what I understand my p-value is in favor of the null hypothesis that $\beta$=0 or similarly that a version of the model with less predictors performs better than the predictors I currently have. I aimed to solve this problem with glmnet to use feature importance. I tried to find the best parameters using caret only to receive the error:

Warning message:
In nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo,  :
There were missing values in resampled performance measures.


From my understanding this is telling me that my $R^2$ statistic is constant over all my values. Please correct me if I am wrong. So from here I used the stepAIC model for feature selection, which actually gives me something tangible:

placenta.model = lm(beta~., data=placenta)
step <- stepAIC(placenta.model, direction="both")

Call:
lm(formula = beta ~ Area + Perimeter + Rmean + UCI_to_Perim +
A_VesselToDiscPercent + A_NumGenerations + A_NumCordBranches +
A_NumBranchPoints + A_NumEndPoints + A_ArcLength + A_StdThickness +
A_MurrayExponent + A_MeanDistToPerim + A_ModeAngle + A_MinAngle +
A_MinTortuosity + V_VesselToDiscPercent + V_ArcLength + V_StdDistToPerim +
V_MeanAngle + V_StdDevAngle + V_MinAngle + V_StdDevTortuosity +
T_.VesselToDiscPercent + T_ArcLength + T_StdDistAEndToVEnd +
T_MeanDistAToV + T_StdDistAToV + A_StdDistToPerim, data = placenta)

Residuals:
Min       1Q   Median       3Q      Max
-0.05454 -0.01598  0.00034  0.01570  0.07043

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)             7.582e-01  2.464e-01   3.077 0.002469 **
Area                   -1.401e-03  5.427e-04  -2.582 0.010740 *
Perimeter               3.194e-03  1.480e-03   2.158 0.032450 *
Rmean                   5.189e-02  2.967e-02   1.749 0.082289 .
UCI_to_Perim           -2.042e-03  1.554e-03  -1.314 0.190696
A_VesselToDiscPercent   6.090e+01  3.893e+01   1.564 0.119745
A_NumGenerations        1.815e-03  1.089e-03   1.667 0.097578 .
A_NumCordBranches      -3.796e-02  1.868e-02  -2.032 0.043838 *
A_NumBranchPoints      -4.785e-02  1.880e-02  -2.545 0.011901 *
A_NumEndPoints          4.670e-02  1.873e-02   2.493 0.013702 *
A_ArcLength             1.142e+02  3.475e+01   3.286 0.001255 **
A_StdThickness         -6.210e-01  2.229e-01  -2.786 0.006006 **
A_MurrayExponent       -1.121e-02  5.619e-03  -1.995 0.047742 *
A_MeanDistToPerim       3.333e-02  1.043e-02   3.194 0.001696 **
A_ModeAngle             1.513e-04  8.068e-05   1.875 0.062609 .
A_MinAngle             -4.074e-04  2.797e-04  -1.457 0.147221
A_MinTortuosity        -5.073e-01  2.189e-01  -2.317 0.021807 *
V_VesselToDiscPercent   6.090e+01  3.893e+01   1.564 0.119788
V_ArcLength             1.142e+02  3.475e+01   3.286 0.001256 **
V_StdDistToPerim        6.415e-02  1.904e-02   3.369 0.000951 ***
V_MeanAngle             7.969e-04  3.466e-04   2.299 0.022832 *
V_StdDevAngle          -3.694e-04  2.835e-04  -1.303 0.194500
V_MinAngle             -4.614e-04  3.186e-04  -1.448 0.149529
V_StdDevTortuosity     -1.032e-01  7.406e-02  -1.394 0.165445
T_.VesselToDiscPercent -6.089e+01  3.893e+01  -1.564 0.119798
T_ArcLength            -1.142e+02  3.475e+01  -3.286 0.001256 **
T_StdDistAEndToVEnd     2.207e-02  1.285e-02   1.717 0.087944 .
T_MeanDistAToV         -1.039e-01  6.254e-02  -1.661 0.098640 .
T_StdDistAToV           1.222e-01  5.427e-02   2.252 0.025703 *
A_StdDistToPerim       -3.116e-02  2.171e-02  -1.436 0.153122
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.02309 on 156 degrees of freedom
Multiple R-squared:  0.3394,    Adjusted R-squared:  0.2167
F-statistic: 2.764 on 29 and 156 DF,  p-value: 3.023e-05


This seems great but I have read in literature that this is not a good way to proceed. So my question here is:

1. When I am trying to fit a linear model and my predictors provider a high p-value what is the best way to proceed? I thought a good way to proceed were by some of the solutions posted above.

2. What do the NAs represent in the first model?

Anything else that anyone can suggest would be great! Thank you!

• When you say your "R^2 statistic is constant over all the values." What values are you referring to? Predictive accuracy after fitting models with any subset of model predictors? – AdamO Nov 2 '15 at 18:54
• @AdamO - check this out - stackoverflow.com/questions/10503784/… – RDizzl3 Nov 2 '15 at 18:59
• I think I see. I suspect the singularity of the design matrix is largely at play. I would encourage you to systematically evaluate your predictors and throw out the ones you know to be bunk. That's the ideal first place to start. – AdamO Nov 2 '15 at 19:08
• @AdamO - that is a good idea. I was also thinking along the lines of feature engineering or interactions between variables. To try to see if I can actually engineer something with a little more predictive power. Of course this would require research on this area to begin with. Good think I'm paired with a biology expert :) – RDizzl3 Nov 2 '15 at 19:10
• I agree strongly with @AdamO. For example, I'd expect Area and Perimeter to be very highly correlated. I'd advise a thorough exploratory data analysis (correlation matrix, scatter plot matrix, consideration of transformations, etc.). While not underplaying the role of domain knowledge, many of the ideas concerned feature in any good first statistics course. – Nick Cox Nov 2 '15 at 19:11

## 1 Answer

Since you are working with nested models, a much better approach might be done with plain LASSO. From what I understand, GLMNET concerns estimating a non-linear relationship between several factors and the outcome of interest, which is generally important if there is one or more factors having a complex non-linear relationship with the outcome. With less than 200 observations and nearly half as many features, the idea of estimating this reliably is ludicrous.

The missing coefficients in your kitchen sink model is a testament to that. The NA values are a consequence of singularity, when one column or more of the model matrix can be expressed as a linear combination of other columns. Therefore if stepwise model selection is the appropriate means of creating a final predictive model, forward stepwise selection is the only viable solution: and iffy at that.

LASSO could be much better. I think you should give that an attempt if this really concerns building a predictive model.

Nonetheless, I can't help but observe that these measurements seem to arise from a somewhat sophisticated physical process. Building predictive models has limited importance in fields where processes are not understood. However, if a process is understood well enough to formulate a model (or two or ten) without relying on complicated machine learning techniques, I wholeheartedly advocate abandoning ML methods in favor of --what I call-- fitting a "physical" model. This is much simpler and easier, usually. Just see the R help entry for nls. Physical model(s) generally provides a fantastic means of communicating data to physicists, biologists, or other groups. Finding 4 or 5 physical models, and comparing AIC provides some means of comparing their predictive accuracy.

• I will try that out! You are correct about the measurements as well. This data is highly specialized to biology centered around child birth and the actual goal itself is to see which measurements are the most important in determining the response I mentioned. Thank you for your response! – RDizzl3 Nov 2 '15 at 19:07