# How do I decide on keeping a variable with a p-value just above alpha?

##### Goal: To find all critical variables that influence time to death so as to not record unneeded variables in future observations.
Call:
coxph(formula = Surv(time, DEATH_EVENT) ~ age + anaemia + creatinine_phosphokinase +
ejection_fraction + serum_creatinine + serum_sodium + hypertension,
data = HF)

coef  exp(coef)   se(coef)      z        p
age                       4.357e-02  1.045e+00  8.831e-03  4.934 8.05e-07
anaemia1                  4.460e-01  1.562e+00  2.150e-01  2.074   0.0380
creatinine_phosphokinase  2.101e-04  1.000e+00  9.825e-05  2.138   0.0325
ejection_fraction        -4.747e-02  9.536e-01  1.027e-02 -4.621 3.82e-06
serum_creatinine          3.139e-01  1.369e+00  6.895e-02  4.552 5.31e-06
serum_sodium             -4.569e-02  9.553e-01  2.336e-02 -1.956   0.0505
hypertensionPresent       4.965e-01  1.643e+00  2.137e-01  2.324   0.0201

Likelihood ratio test=80.58  on 7 df, p=1.048e-14
n= 299, number of events= 96


This above selection of variables was derived from performing backward stepwise selection. 3 other variables were eliminated.

If you look at serum_sodium, you'll see a p-value of 0.0505 which is just over alpha = 0.05. I've never run into a situation in which the p-value is that close to alpha.

What steps are usually taken to decide on whether to include it or not?

#### Update

I performed a likelihood ratio test between the models and the null hypothesis, both models perform equally well, was not rejected. In the end, I dropped the variable but would appreciate responses had I not chosen this route.

• You need to set a strict level to reject your hypothesis before you do any kind of analysis. Once you determined your criterion then you must stick with it. Even if something is just on the border. Lines need to be drawn somewhere and this is the criterion you decided to set initially. Dec 17, 2022 at 22:43
• There’s something to what @NicolasBourbaki wrote about sticking to your guns about your cutoff, lest you wind up in a situation where $0.051$ is so close that you reject and $0.049$ is so close that you don’t reject (among other reasons). However, why screen variables this way at all? What are you hoping to accomplish? Screening variables based on p-values distorts all downstream inferences, including pure predictive ability, unless you take considerable care that most do not.
– Dave
Dec 17, 2022 at 22:48
• – whuber
Dec 17, 2022 at 23:12
• My suggestion is to say what you hope to accomplish by screening variables based on p-values and how such screening helps you accomplish your objectives.
– Dave
Dec 17, 2022 at 23:13
• Were these 7 predictors the only ones that you evaluated in this model, or does that number represent some prior predictor selection based on outcomes? Please edit the question to address that, to say what your goal is in building this model, and to describe why you think that "screening variables based on p-values" will bring you closer to your goal. Please do that by editing the question, as comments are easy to overlook and can be deleted.
– EdM
Dec 18, 2022 at 14:12

Unless you need a parsimonious model with few predictors there is seldom a reason to omit an outcome-associated predictor from a model, provided that the model isn't overfit. Backward stepwise selection might be one of the least objectionable among the unreliable methods for automated variable selection, but that still doesn't often make it a good choice.

Since you did backward selection to arrive at the model you display, you should recognize that the p-values are already invalid. The calculation of those p-values assumes that the final model was built independently of seeing the outcomes. But that's not what you did. That makes any decision based upon a post-selection p-value even more unreliable. Look at the links suggested in a comment by @whuber for many more details.

Predictor elimination is particularly dangerous with Cox regression, which has an omitted-variable bias similar to what's found with binary regression. Even if an omitted outcome-associated predictor is uncorrelated with the included predictors, keeping it out of a Cox model can bias the estimates of the coefficients for the included predictors.

The best approach is to use your understanding of the subject matter to build a model as complex as reasonable while unlikely to overfit. Frank Harrell's course notes and book provide extensive guidance on how to proceed when you seem to have too many predictors for a limited amount of data. See Chapter 4 of each, in particular. Then, unless "parsimony is more important than accuracy," report the model results as is, without pruning the number of predictors.

In the question you say that you want to "find all critical variables that influence time to death so as to not record unneeded variables in future observations," which suggests that parsimony is important to you. But does it really make sense not to keep track of a common and easy-to-obtain predictor like serum sodium? With a Cox model, err on the side of inclusion.

• Genuine question. Does this mean that after even just the first dropped variable (using -p-values), the p-values in the subsequent model become less valid? Dec 18, 2022 at 23:43
• @Antonio yes, unless you take extra precautions. The p-value calculation done by coxph() assumes that you have a pre-specified model and that you haven't used prior results to alter the model. Once you use results to alter the model, that assumption is violated. It's possible to evaluate the magnitude of such a model's overfitting "optimism" with bootstrapping. The validate() function in the R rms package combines that with backward elimination, if you really need backward elimination.
– EdM
Dec 19, 2022 at 12:55