1
$\begingroup$

So I am trying to work out what is the best predictor of a) awareness over environmental issues, b) concern over environmental issues and c) pro-environmental behaviour from a set of sociodemographics (eg. age bracket, political standing, location etc) measured using a survey.

I am using a backward stepwise linear regression to do this. I get clear results when I input all the sociodemographics apart from location. There are 7 predictors for model a, 4 predictors for model b and 1 predictor for model c. However, when I include location as a possible predictor, it excludes almost none of the other 12 possible predictors in the output for each of the 3 models.

I am wondering whether this is something to do with the high number of variables within the location category - there were 164 respondents from 82 different locations. However, with every other possible predictor eg. political standing, there were roughly 6 variable groups reported by participants in the survey.

If anyone had any indication of what might be going on and any advice on whether to leave location out of the analysis it would be much appreciated! (Also any advice on how I would justify excluding location in my methods would be amazing!)

$\endgroup$

1 Answer 1

0
$\begingroup$

There are several ways to see if your more complex model is statistically justified compared to a more simple model.

First, you can check AIC values, keeping the model with the lowest AIC value as "the best model".

Second, you could do an ANOVA between the most complex model, compared to the subsequent simpler ones, and make a judgment based on the resulting p.value from the ANOVA.

You can also perform log-likelihood ratio tests, and just like with the ANOVA, it will give you a p.value which can help you decide whether the more complex model is justified or not.

You can also compare the r-squared values from each model (see if the simpler model improves the r-sqaured value, or vice versa)

Finally, you could also see if your model meets the correct assumptions with quantile-quantile plots (qqplots), and sometimes, I've seen qqplots get better with simpler models (and vice versa). However, I would caution this last step, since the other three I listed are published methods on how to verify model selection.

As far as why location might be causing you issues: I could see it being an issue that it has 82 different levels (i.e. locations)! That's a lot of different things to predict in a model, and you risk just fitting whats called "noise". You could maybe make the variable more simple, and have groups such as "north, south, east, west", from the location data. Another thing you could consider is running such a variable as a random factor, but this would assume that you are not interested in "location" directly influencing your data, but it is nonetheless important information in keeping in the model. It will standardize the effect of location across all explanatory variables, and still let you see how large the effects are from your explanatory variables after correction. Lastly, don't forget that if a variable is statistically significant, this is being compared to the model intercept, and not other variables in the model! If you are interested in such information, you need to then continue with subsequent pairwise testing.

Following our discussion in the comment thread about AIC, ANOVA, etc.., here is an example with the mtcarsdataset that you could try yourself.

data("mtcars")

df <- mtcars

?mtcars() # gives you info about the dataset

names(df) # we have 11 columns of data, lets pick just 3 to see if miles per gallon (mpg) is explained by:

#       1) cyl : # of cylinders
#       2) hp : horse power
#       3) drat : rear axle ratio

attach(df)

full_model <-lm(mpg~cyl+hp+drat)
summary(full_model)

enter image description here

We can see from the above image that none of the predictor variables are significant, so let's remove the one that has the highest p.value (in this case, cyl) and re-run the model

reduced_model <-lm(mpg~hp+drat)
summary(reduced_model)

enter image description here

Now we have some statistically significant predictors, but should we have use the reduced model? Let's check the AIC scores, as well as run an ANOVA between the models, and finally, compare R^2 values

AIC(full_model,reduced_model)
anova(full_model,reduced_model)

Despite certain predictors now becoming statistically significant, our AIC value is lower for the more complex model, suggesting we should use the more complex model

enter image description here

What about the ANOVA comparison?

enter image description here

Here, we see that we don't have a very high F-Value between the models, suggesting the difference between the two is not very large, and the p.value is not below our threshold of 0.05, so it is not significant. Also, we see a degree of freedom of -1 for our reduced model, verifying that we did indeed compare one model that has 1 less variable than the other. Because our result is not significant, we would reject model 2, our "reduced_model" because there is not evidence that it improved it, and keep our most complex model.

Finally, if we look at the R^2 value, we see that the full_model has a slightly higher value.

So three indicators that our more complex model, here, is the better one.

I just made this problem up as I was typing, and know that it also could have issues, such as correlation issues between the number of cylinders and horsepower. That's a different avenue to explore. But, the point was to show how you can do model selection, and "verify" if the simpler model is more justified than the more complex one. Here, all indicators suggest the more complex one is the way to go.

$\endgroup$
12
  • $\begingroup$ How do you figure that the simpler model could have a higher $R^2?$ Do you mean some kind of out-of-sample performance? $\endgroup$
    – Dave
    Apr 11, 2022 at 18:01
  • 1
    $\begingroup$ I don't believe locations are being predicted: they are being used as (fixed effect) explanatory variables. That's too many parameters. You should express scepticism about the meaning of such a model and, to address the last part of the question, offer alternatives such as a random effects model. A few words about the problems with using stepwise regression in this setting might be helpful, too. $\endgroup$
    – whuber
    Apr 11, 2022 at 18:42
  • $\begingroup$ @Dave: 𝑅2 , just in case this isn't clear, explains how well the data fit the model, and subsequently, how much variation is explained by the model. An R2 of 1 would equal a perfect fit, meaning we explain 100% of the data. R2=.45, means we can account for 45% of the variation in the dataset. If you look at the model summaries between your models, you can see how the R2 value changes, and make a judgement which model is better. BUT, simpler does not mean better, keep that in mind. It's possible the more complex answer is better. If you like my answer, will you accept it please :) ! $\endgroup$
    – Andy
    Apr 12, 2022 at 11:26
  • $\begingroup$ Thanks so much for your response! It's incredibly appreciated. So I have looked at the AIC values between the two sets of 3 and the more complex models have a lower AIC for all. I wondered how doing an ANOVA and the resulting p values would tell me which one is statistically justified? Could you explain slightly more please? I will also have a go at making the locations slightly more simple and see if this works better. Thank you! $\endgroup$
    – Max Clarke
    Apr 12, 2022 at 12:24
  • $\begingroup$ @MillieClarke an anova between models is easy (I code with R, so the following will be in R language, but it can be adapted to whatever you are using). Let's say I have two models, one with three variables, the other with two variables. complex_model <- lm(response~x1 + x2 + x3) , and simpler_model <- lm(response~x1+x2). To check an ANOVA between the two, just run anova( complex_model,simpler_model) If the resulting p.value is not significant, this means there is no evidence to keep the more complex model, and vice versa $\endgroup$
    – Andy
    Apr 12, 2022 at 14:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.