Does my predictor in my multiple regression have too many variables? So I am trying to work out what is the best predictor of a) awareness over environmental issues, b) concern over environmental issues and c) pro-environmental behaviour from a set of sociodemographics (eg. age bracket, political standing, location etc) measured using a survey.
I am using a backward stepwise linear regression to do this. I get clear results when I input all the sociodemographics apart from location. There are 7 predictors for model a, 4 predictors for model b and 1 predictor for model c. However, when I include location as a possible predictor, it excludes almost none of the other 12 possible predictors in the output for each of the 3 models.
I am wondering whether this is something to do with the high number of variables within the location category - there were 164 respondents from 82 different locations. However, with every other possible predictor eg. political standing, there were roughly 6  variable groups reported by participants in the survey.
If anyone had any indication of what might be going on and any advice on whether to leave location out of the analysis it would be much appreciated! (Also any advice on how I would justify excluding location in my methods would be amazing!)
 A: There are several ways to see if your more complex model is statistically justified compared to a more simple model.
First, you can check AIC values, keeping the model with the lowest AIC value as "the best model".
Second, you could do an ANOVA between the most complex model, compared to the subsequent simpler ones, and make a judgment based on the resulting p.value from the ANOVA.
You can also perform log-likelihood ratio tests, and just like with the ANOVA, it will give you a p.value which can help you decide whether the more complex model is justified or not.
You can also compare the r-squared values from each model (see if the simpler model improves the r-sqaured value, or vice versa)
Finally, you could also see if your model meets the correct assumptions with quantile-quantile plots (qqplots), and sometimes, I've seen qqplots get better with simpler models (and vice versa).  However, I would caution this last step, since the other three I listed are published methods on how to verify model selection.
As far as why location might be causing you issues: I could see it being an issue that it has 82 different levels (i.e. locations)!  That's a lot of different things to predict in a model, and you risk just fitting whats called "noise". You could maybe make the variable more simple, and have groups such as "north, south, east, west", from the location data.  Another thing you could consider is running such a variable as a random factor, but this would assume that you are not interested in "location" directly influencing your data, but it is nonetheless important information in keeping in the model.  It will standardize the effect of location across all explanatory variables, and still let you see how large the effects are from your explanatory variables after correction.  Lastly, don't forget that if a variable is statistically significant, this is being compared to the model intercept, and not other variables in the model! If you are interested in such information, you need to then continue with subsequent pairwise testing.
Following our discussion in the comment thread about AIC, ANOVA, etc.., here is an example with the mtcarsdataset that you could try yourself.
data("mtcars")

df <- mtcars

?mtcars() # gives you info about the dataset

names(df) # we have 11 columns of data, lets pick just 3 to see if miles per gallon (mpg) is explained by:

#       1) cyl : # of cylinders
#       2) hp : horse power
#       3) drat : rear axle ratio

attach(df)


full_model <-lm(mpg~cyl+hp+drat)
summary(full_model)


We can see from the above image that none of the predictor variables are significant, so let's remove the one that has the highest p.value (in this case, cyl) and re-run the model
reduced_model <-lm(mpg~hp+drat)
summary(reduced_model)


Now we have some statistically significant predictors, but should we have use the reduced model?  Let's check the AIC scores, as well as run an ANOVA between the models, and finally, compare R^2 values
AIC(full_model,reduced_model)
anova(full_model,reduced_model)

Despite certain predictors now becoming statistically significant, our AIC value is lower for the more complex model, suggesting we should use the more complex model

What about the ANOVA comparison?

Here, we see that we don't have a very high F-Value between the models, suggesting the difference between the two is not very large, and the p.value is not below our threshold of 0.05, so it is not significant.  Also, we see a degree of freedom of -1 for our reduced model, verifying that we did indeed compare one model that has 1 less variable than the other.  Because our result is not significant, we would reject model 2, our "reduced_model" because there is not evidence that it improved it, and keep our most complex model.
Finally, if we look at the R^2 value, we see that the full_model has a slightly higher value.
So three indicators that our more complex model, here, is the better one.
I just made this problem up as I was typing, and know that it also could have issues, such as correlation issues between the number of cylinders and horsepower. That's a different avenue to explore.  But, the point was to show how you can do model selection, and "verify" if the simpler model is more justified than the more complex one.  Here, all indicators suggest the more complex one is the way to go.
