What you propose is not the way I would choose which variables to include in a model. In my own opinion and practice, I prefer to determine the number of independent variables I am willing to include in my model before I ever start programming the model.
I will generally only accept one degree of freedom to be used up for every ten events I've observed in my data. If I have 100 events, I will accept no more than ten degrees of freedom to be used. If I have 20 events, I will accept no more than two degrees of freedom to be used. Keep in mind that categorical variables may use more than one degree of freedom. A three category variable will use two degrees of freedom, for example. And obviously, the more events per degree of freedom, the better. The 10-to-1 rule of thumb (and I honestly couldn't tell you where it comes from) is just a way to balance the need for adequate fit with including more variables.
After I know how many degrees of freedom I'll include, my next priority is to have a subject matter expert rank the available variables in order of likely influential variables. It's a subjective call, but if I'm limited in how many variables I can include, I want to try to use the most meaningful ones for subject matter experts.
Beyond that, I have no real process for deciding. I'll look at multicollinearity with Variance Inflation Factors. If I find VIFs larger than ten, I may consider removing one of the collinear variables (caveat: unless I'm interested in predictive accuracy. Multicollinearity is less of an issue in prediction than it is in identification of risk factors. But if you allow multicollinearity into your model, be careful not to try to put too much faith in the coefficients).
If I really want to consider removing a variable, I'm more likely to use a likelihood ratio test between a model with the variable I'm considering and the model without it. If the likelihood ratio test shows statistical significance, then I conclude that the variable is important in the model and keep it. If it doesn't show statistical significance, I feel less concerned about removing it (though it still comes with risks).
Another approach to take is to remove the variable and see if the other coefficients change much. ("much" isn't a well defined word here). If I notice big changes in the coefficients without the variable, I'll keep it. Otherwise, I'm okay removing it.
These approaches have their weaknesses, and they can be thoroughly criticized. Hopefully some smarter people will chime in on other things to consider.