Methodology for calculating variable importance in dataset using regression

Question

I am trying to come up with an understanding of the magnitude of the effect of various variables on a (measurable) continuous target variable from data based on a survey. There are 2000 variables consisting of survey questions as well as some index and socioeconomic variables. Both categorical and continuous variables are present.

All of the variables are actually grouped into 1 of 5 categories. Consider these as question types. Ideally, I would like to have a chart that breaks down in percentage terms the importance of question type 1 on the target, etc. The breakdown for the 5 categories would add up to 100%

Here is my idea.

1. For each categorical variable, make sure that the base level in the factor (using R) makes logical sense. Most of the categorical questions have been pre-processed so that the missing, unavailable, not administered "answers" to the questions are labeled as NA. However, this is still considered an answer option in the analysis as opposed to a missing value, because there may be valuable information from this. This "NA" designation can be set as the base level for many of the questions.
2. Run a multiple linear regression on the raw data.
3. Check for multi-collinearity and re-run, excluding the variables with high VIFs.
4. Examine the distribution of questions in each category. Create weights to ensure that the actual distribution of categorial + continuous questions in each category is roughly equal. Otherwise, if 1 category tends to have more questions, the purported impact will be skewed. It may make sense to use only the number of categorical questions in each category for the weights.
5. Normalize the coefficients first by dividing by the variance of the variable. This is performed because of course the variables will have both positive and negative effects on the target variable.
6. Standardize the coefficients by subtracting the value of the minimum coefficient and then dividing by the range of the coefficients.
7. Multiply each coefficient by its weight and then take the sum.
8. Express each (variable, weight) product as a percentage of the total in the step above.
9. Group the questions according to the categories lookup.

Is this a valid approach? What are the pitfalls and major assumptions? Because there are so many variables, I am ignoring p-values completely. Is that appropriate?

Any suggestions or ideas on an alternative approach welcome. Thank you.

Answer 1

There is a substantial literature on relative importance so I would suggest using an established technique. In her paper "Estimators of relative importance in linear regression based on variance decomposition" here Grömping suggests criteria for choosing between methods and then gives a detailed comparison between two methods: LMG which obtains the Model I sums of squares for each predictor and then averages over all the possible orders of fitting the predictors, and PMVD which is a weighted version of LMG. In an earlier paper discussing her software "Relative importance for linear regression in R: the package relaimpo" here she also discusses four other versions.