17
$\begingroup$

This is my first post on StackExchange, but I have been using it as a resource for quite a while, I will do my best to use the appropriate format and make the appropriate edits. Also, this is a multi-part question. I wasn't sure if I should split the question into several different posts or just one. Since the questions are all from one section in the same text I thought it would be more relevant to post as one question.

I am researching habitat use of a large mammal species for a Master's Thesis. The goal of this project is to provide forest managers (who are most likely not statisticians) with a practical framework to assess the quality of habitat on the lands they manage in regard to this species. This animal is relatively elusive, a habitat specialist, and usually located in remote areas. Relatively few studies have been carried out regarding the distribution of the species, especially seasonally. Several animals were fitted with GPS collars for a period of one year. One hundred locations (50 summer and 50 winter) were randomly selected from each animal's GPS collar data. In addition, 50 points were randomly generated within each animal's home range to serve as "available" or "pseudo-absence" locations. The locations from the GPS collars are coded a 1 and the randomly selected available locations are coded as 0.

For each location, several habitat variables were sampled in the field (tree diameters, horizontal cover, coarse woody debris, etc) and several were sampled remotely through GIS (elevation, distance to road, ruggedness, etc). The variables are mostly continuous except for 1 categorical variable that has 7 levels.

My goal is to use regression modelling to build resource selection functions (RSF) to model the relative probability of use of resource units. I would like to build a seasonal (winter and summer) RSF for the population of animals (design type I) as well as each individual animal (design type III).

I am using R to perform the statistical analysis.

The primary text I have been using is…

  • "Hosmer, D. W., Lemeshow, S., & Sturdivant, R. X. 2013. Applied Logistic Regression. Wiley, Chicester".

The majority of the examples in Hosmer et al. use STATA, I have also been using the following 2 texts for reference with R.

  • "Crawley, M. J. 2005. Statistics : an introduction using R. J. Wiley, Chichester, West Sussex, England."
  • "Plant, R. E. 2012. Spatial Data Analysis in Ecology and Agriculture Using R. CRC Press, London, GBR."

I am currently following the steps in Chapter 4 of Hosmer et al. for the "Purposeful Selection of Covariates" and have a few questions about the process. I have outlined the first few steps in the text below to aid in my questions.

  1. Step 1: A univariable analysis of each independent variable (I used a univariable logistic regression). Any variable whose univariable test has a p-value of less than 0.25 should be included in the first multivariable model.
  2. Step 2: Fit a multivariable model containing all covariates identified for inclusion at step 1 and to assess the importance of each covariate using the p-value of its Wald statistic. Variables that do not contribute at traditional levels of significance should be eliminated and a new model fit. The newer, smaller model should be compared to the old, larger model using the partial likelihood ratio test.
  3. Step 3: Compare the values of the estimated coefficients in the smaller model to their respective values from the large model. Any variable whose coefficient has changed markedly in magnitude should be added back into the model as it is important in the sense of providing a needed adjustment of the effect of the variables that remain in the model. Cycle through steps 2 and 3 until it appears that all of the important variables are included in the model and those excluded are clinically and/or statistically unimportant. Hosmer et al. use the "delta-beta-hat-percent" as a measure of the change in magnitude of the coefficients. They suggest a significant change as a delta-beta-hat-percent of >20%. Hosmer et al. define the delta-beta-hat-percent as $\Delta\hat{\beta}\%=100\frac{\hat{\theta}_{1}-\hat{\beta}_{1}}{\hat{\beta}_{1}}$. Where $\hat{\theta}_{1}$ is the coefficient from the smaller model and $\hat{\beta}_{1}$ is the coefficient from the larger model.
  4. Step 4: Add each variable not selected in Step 1 to the model obtained at the end of step 3, one at a time, and check its significance either by the Wald statistic p-value or the partial likelihood ratio test if it is a categorical variable with more than 2 levels. This step is vital for identifying variables that, by themselves, are not significantly related to the outcome but make an important contribution in the presence of other variables. We refer to the model at the end of Step 4 as the preliminary main effects model.
  5. Steps 5-7: I have not progressed to this point so I will leave these steps out for now, or save them for a different question.

My questions:

  1. In step 2, what would be appropriate as a traditional level of significance, a p-value of <0.05 something larger like <.25?
  2. In step 2 again, I want to make sure the R code I have been using for the partial likelihood test is correct and I want to make sure I am interpreting the results correctly. Here is what I have been doing…anova(smallmodel,largemodel,test='Chisq') If the p-value is significant (<0.05) I add the variable back to the model, if it is insignificant I proceed with deletion?
  3. In step 3, I have a question regarding the delta-beta-hat-percent and when it is appropriate to add an excluded variable back to the model. For example, I exclude one variable from the model and it changes the $\Delta\hat{\beta}\%$ for a different variable by >20%. However, the variable with the >20% change in $\Delta\hat{\beta}\%$ seems to be insignificant and looks as if it will be excluded from the model in the next few cycles of Steps 2 and 3. How can I make a determination if both variables should be included or excluded from the model? Because I am proceeding by excluding 1 variable at a time by deleting the least significant variables first, I am hesitant to exclude a variable out of order.
  4. Finally, I want to make sure the code I am using to calculate $\Delta\hat{\beta}\%$ is correct. I have been using the following code. If there is a package that will do this for me or a more simple way of doing it I am open to suggestions.

    100*((smallmodel$coef[2]-largemodel$coef[2])/largemodel$coef[2])

$\endgroup$
  • $\begingroup$ out of curiosity what is the species that you are studying ? $\endgroup$ – forecaster Oct 13 '14 at 21:33
23
$\begingroup$

None of those proposed methods have been shown by simulation studies to work. Spend your efforts formulating a complete model and then fit it. Univariate screening is a terrible approach to model formulation, and the other components of stepwise variable selection you hope to use should likewise be avoided. This has been discussed at length on this site. What gave you the idea in the first place that variables should sometimes be removed from models because they are not "significant"? Don't use $P$-values or changes in $\beta$ to guide any of the model specification.

$\endgroup$
  • 3
    $\begingroup$ Yes, domain knowledge + a healthy dose of disbelief in simplicity, e.g., don't assume continuous variables act linearly unless you have prior data demonstrating linearity. $\endgroup$ – Frank Harrell Oct 8 '14 at 10:41
  • 6
    $\begingroup$ The OP is citing a mainstream text in its third edition with authors who have made great contributions to the field. Other points made in the question are discussed in other influential texts (Agresti, Gelman). I bring this up not because I agree with this strategy, but rather to note that these strategies are advised in recent, mainstream texts by respected statisticians. In sum: although there is plenty of literature advising against this, it does not seem to be rejected by the statistical community. $\endgroup$ – julieth Oct 10 '14 at 12:57
  • 2
    $\begingroup$ That is quite misguided in my humble opinion. The strategies pushed so hard in some texts have never been validated. Authors who do not believe in simulation put themselves at risk for advocating the use of methods that do not work as advertised. $\endgroup$ – Frank Harrell Oct 10 '14 at 16:20
  • 2
    $\begingroup$ Yes, I know. I refer to your text and papers often, and its one of the sources I have used to arrive at my conclusion disagreeing with the above strategy. I am simply conveying the dilemma of the applied user. We cannot test everything. We rely on experts, such as you. $\endgroup$ – julieth Oct 10 '14 at 20:08
  • 3
    $\begingroup$ @GNG: FH is referring to simulation as a way of showing that this approach to model selection actually does what it's supposed to do (presumably to improve the accuracy of your model's predictions) in typical applications. Your (astute) questions highlight its rather arbitrary, ad hoc, nature - basing variable inclusion on an indeterminate number of significance tests at "traditional" levels can't be shown by theory to guarantee the optimization of anything. $\endgroup$ – Scortchi Oct 13 '14 at 10:47
5
$\begingroup$

Methods specified for variable selection using statistic such as P, stepwise regression in the classic text Hosmer et al should at all cost be avoided.

Recently I stumbled upon an article that was published in the international journal of forecasting entitle "Illusions of predictability" and a commentory on this article by Keith ord. I would highly recommend both these article as they clearly show that using regression statistic is often misleading. Follwoing is a screenshot of Keith Ord's article that shows by simulation why step wise regression (uses p statistic) for variable selection is bad.

enter image description here

Another wonderful article by Scott Armstrong that appeared in the same issue of the journal shows why one should be very cautious on using regression analysis on non-experimental data with case studies. Ever since I read these articles I avoid using regression analysis to draw causal inferences on non-experimental data. As a practitioner, I wish I had read articles like this many years which would have saved me from making bad decisions and avoiding costly mistakes.

On your specific problem, I don't think randomized experiments are possible in your case, so I would recommend that you use cross validation to select variables. A nice worked out example is available in this free online book on how you would use predictive accuracy to select variables. It also many othervariable selction methods, but I woud restrict to cross validation.

I personally like the quote from Armstrong "Somewhere I encountered the idea that statistics was supposed to aid communication. Complex regression methods and a flock of diagnostic statistics have taken us in the other direction"

Below is my own opinion. I'm not a statistician.

  • As a biologist I think you would appreciate this point. Nature is very complex, assuming logistic function and no interaction among variables does not occur in nature. In addition, logistic regression has following assumptions:

  • The true conditional probabilities are a logistic function of the independent variables.

  • No important variables are omitted. No extraneous variables are included.

  • The independent variables are measured without error.
  • The observations are independent.
  • The independent variables are not linear combinations of each other.

I would recommend classification and regression tree (CART(r)) as an alternative over logistic regression for this type of analysis because it is assumptions free:

  1. Non parametric/Data Driven/No assumptions that your output probablities follow logistic function.
  2. Non linear
  3. allows complex variable interaction.
  4. Provides highly interpretable visual trees that a non statistician like forest managers would appreciate.
  5. Easily handles missing values.
  6. Dont need to be a statistician to use CART!!
  7. automatically selects variables using cross validation.

CART is a trademark of Salford Systems. See this video for introduction and history of CART. There are also other videos such as cart - logistic regrssion hybrids in the same website. I would check it out. an open source impentation in R is called Tree, and there are many other packages such as rattle available in R. If I find time, I will post the first example in Homser's text using CART. If you insist on using logistic regression, then I would at least use methods like CART to select variables and then apply logistic regression.

I personally prefer CART over logistic regression because of aforementioned advantages. But still, I would try both logistic regression and CART or CART-Logistc Regression Hybrid, and see which gives better predictive accuracy and also more importantly better interpretatablity and choose the one that you feel would "communicate" the data more clearly.

Also, FYI CART was rejected by major statistical journals and finally the inventors of CART came out with a monograph. CART paved way to modern and highly successful machine learning algorithms like Random Forest(r), Gradient Boosting Machines (GBM), Multivariate Adaptive Regression Splines all were born. Randomforest and GBM are more accurate than CART but less interprettable (black box like) than CART.

Hopefully this is helpful. Let me know if you find this post useful ?

$\endgroup$
  • 8
    $\begingroup$ No. The logistic model does not make more assumptions than other models. It's main unique assumption is that $Y$ is truly all-or-nothing. CART is hugely outperformed by logistic regression. CART effectively fits far more parameters than logistic regression because it allows for all possible interactions. The irony is that a method that allows maximum flexibility is more conservative than a more structured method. You'll find that in order for CART models to be well-calibrated you have to prune the model down to have small predictive discrimination. $\endgroup$ – Frank Harrell Oct 11 '14 at 3:03
  • 3
    $\begingroup$ This answer jumps from general comments, many of which seem uncontroversial at least to me, to a highly specific and rather personal endorsement of CART as the method of choice. You're entitled to your views, as others will be entitled to their objections. My suggestion is that that you flag the twofold flavour of your answer rather more clearly. $\endgroup$ – Nick Cox Oct 13 '14 at 17:59
  • 2
    $\begingroup$ Logistic regression is a generalised linear model, but otherwise it is defensible as, indeed well motivated as, a naturally nonlinear model (in the sense that it fits curves or equivalent, not lines or equivalent, in the usual space) that is well suited to binary responses. The appeal to biology here is double-edged; historically logistic models for binary responses were inspired by models for logistic growth (e.g. of populations) in biology! $\endgroup$ – Nick Cox Oct 13 '14 at 18:01
  • $\begingroup$ The Soyer et al. paper, the Armstrong paper, and commentaries are all very good. I have been reading over them this weekend. Thank you for suggesting them. Not being a statistician I cannot comment on using CART over logistic regression. However, your answer is very well written, helpful, and has received comments that are insightful. I have been reading up on machine learning methods such as CART, MaxEnt, and boosted regression trees and am planning on discussing them with my committee to get their insight. When I get some free time, the CART video should be interesting as well. $\endgroup$ – GNG Oct 13 '14 at 18:18
  • 3
    $\begingroup$ With a smile I think we can reverse your comments on linear models and insist that far from being assumption-free, or even assumption-light, CART assumes that reality is like a tree (what else?). If you think that nature is a smoothly varying continuum you should run in the opposite direction. $\endgroup$ – Nick Cox Oct 13 '14 at 21:27
3
$\begingroup$

I think you're trying to predict the presence of the species with a presence/background approach, which is well documented in journals such as Methods in Ecology and Evolution, Ecography, etc. Maybe the R package dismo is useful for your problem. It includes a nice vignette. Using the dismo or other similar package implies to change your approach to the problem, but I believe it's worth to have a look at.

$\endgroup$
  • 2
    $\begingroup$ What keeps you from just specifying a model? Why the great uncertainty in what should be in the model? Why the need for model selection using GLM? $\endgroup$ – Frank Harrell Oct 10 '14 at 20:05
  • 1
    $\begingroup$ I'm afraid you're mixing some concepts. (1) in fact maxent is a presence/background data, or presence/pseudo-absence data. So, maxent uses the presence-only data and adds some points from the landscape, that is, the background/pseudo-absences. Thus, it can be used in your case. (2) GLM were designed to be used with 'true' absences. However, GLM has been adapted for presence/pseudo-absence data. (3) dismo package offers boosted regression trees but not only. You can fit GLM as well, just follow one of the package's vignettes (there are 2). $\endgroup$ – Hugo Oct 10 '14 at 20:55
  • 1
    $\begingroup$ If your question is about which variables you should include as predictors, take a look at these papers: Sheppard 2013. How does selection of climate variables affect predictions of species distributions? A case study of three new weeds in New Zealand. Weed Research; Harris, et al. 2013. To Be Or Not to Be? Variable selection can change the projected fate of a threatened species under future climate. Ecol. Manag. Restor. $\endgroup$ – Hugo Oct 10 '14 at 20:58
  • 2
    $\begingroup$ The thought that variable selection techniques somehow reduce overfitting is strange. The apparent savings of variables from reducing the model is completely an illusion when the reduction comes from the data themselves. $\endgroup$ – Frank Harrell Oct 11 '14 at 3:00
  • 1
    $\begingroup$ @GNG: "My uncertainty about leaving all of the variables in the model comes from everything I have been taught about collinearity and over-fitting" - Does your model contain highly collinear predictors? Is your model over-fitting? $\endgroup$ – Scortchi Oct 13 '14 at 20:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.