How do I select predictors for my quantile regression model?

Question

I'm using a housing dataset that contains predictors such as # of bedrooms, bathrooms, car spots, distance from city center etc. to predict the price of sold houses. I'll be performing quantile regression on my dataset on the 25th, 50th, and 75th percentiles, and was curious how one goes about selecting the variables.

My query: Does one perform OLS and then use the predictors that were significant for the OLS model to look at how those predictors perform at the three quantiles, or does one perform the quantile regression and then remove variables that are insignificant at all three quantiles? Or is there another recommended selection method entirely?

I've looked around Google using various keywords (i.e. "variable selection for quantile regression", "quantile regression how to", etc. and many of them cover regularized techniques like LASSO-quantile regression rather just regular selection for quantile regression). Thanks for the guidance!

Answer 1

Selecting variables based on a p-value significance is generally a bad idea. You can see that by looking at the formulas of the test statistic in a simple linear regression setting (for illustrative purposes): the $t$ statistic for testing wether $\beta_i$ is 0: $$ \frac{\hat{\beta_i}}{\hat{\textrm{se}}(\hat{\beta_i})}$$ And, $$\hat{\textrm{se}}(\hat{\beta_i}) = \sqrt{\frac{\hat{\sigma^2}}{n}(\frac{1}{n}\mathbf{x}^T\mathbf{x})^{-1}}$$

Now, given that in a single dataset $n$ and $\sigma$ are the same, the things that will make the test statistic larger, and as a result more significant are:

1. Magnitude of $|\beta_i|$ (the larger the more significant)
2. The sample variance of $X_i$ (when larger it makes the standard error smaller thus increasing the test statistic).

This is assuming that there is no coliniarity between predictors. If there is, the denominator of the test statistic formula will get an extra addition and when the correlation is large, all else being equal, this will tend to decrease the test statistic and thus the significance.

So we can see that the most significant variables are not necessarily the most "important" ones, they may be just measured more precisely.

Furthermore, you can see that significance also depends on the design of your predictor matrix , so you can have two people with slightly different designs for $\mathbf{x}$ dealing with the same dataset that will get different sets of significant coefficients! This is because adding or removing variables affects the significance through both changing $\beta_i$ and the standard error.

One additional point is that if you could increase the number of observations in your dataset (everything else staying the same), this thing alone can increase the significance of your variables. If $\beta_i$ is not exactly zero, the t-statistic will tend to grow towards infinity as the number of observations increases.

You can state several models that you think, based on theory and other domain-specific considerations, will best describe your dependent variable and then use methods like the AIC to select the best model among them. Note that if you do that you should first split your data into two parts. One of them you are going to use to select the best model specification, and the other will be used to estimate the coefficients. That is the proper way.