High R-squared although many insignificant coefficients

Question

I just did a regression based on the gravity model where I try to identify the most important factors that determine the trade flows. In total I have 18 variables and 363 observations. In fact I would do several regressions not including all variables at once. Although I included them all just to have a look and I got an R-squared of 0,9162 (using robust standard errors). Due to poor data availability for certain variables I only have 102 observations by including all variables. The problem is that only 9 coefficients are significant at the 10% level. Can I now assume that only those variables have a significant impact on trade flows? Can I assume that the regression is good as R-squared is quite high? Or can I do some tests whether those results are reliable (so that I can use them in my research paper)?

I am doing this for the first time and I am a little bit confused.

Answer 1

There's some chance you're overfitting the data with this sample size (the $R^2$ value seems suspiciously high), but more to the point there's nothing inconsistent between a high $R^2$ and lots of "insignificant" predictors. This is simply because the coefficient of determination never decreases when you add variables to the model, so if you start with a high $R^2$ then you'll end up with one as long as you don't drop any variables. Below are some simple simulations that demonstrate this idea. Here I generated data according to the model $$ Y_i = x_{1i} + \epsilon_i $$ where $\epsilon_i \sim$ normal$(0, \sigma^2)$ and then fit a regression only involving $x_{1i}$ along with another that included nine extra noise predictors $x_{2i}, x_{3i}, \ldots, x_{10i}$ that bore no relation to $Y_i$. We can see that the second model has a larger $R^2$ even though the added variables are not important.

Model 1:

                            OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.935
Model:                            OLS   Adj. R-squared:                  0.934
Method:                 Least Squares   F-statistic:                     1424.
Date:                Sun, 02 Aug 2015   Prob (F-statistic):           1.44e-60
Time:                        22:37:24   Log-Likelihood:                -4.2454
No. Observations:                 100   AIC:                             10.49
Df Residuals:                      99   BIC:                             13.10
Df Model:                           1                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
var_1          0.9865      0.026     37.734      0.000         0.935     1.038
==============================================================================
Omnibus:                        7.358   Durbin-Watson:                   1.957
Prob(Omnibus):                  0.025   Jarque-Bera (JB):                3.027
Skew:                           0.016   Prob(JB):                        0.220
Kurtosis:                       2.148   Cond. No.                         1.00
==============================================================================

Model 2:

                            OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.941
Model:                            OLS   Adj. R-squared:                  0.934
Method:                 Least Squares   F-statistic:                     143.2
Date:                Sun, 02 Aug 2015   Prob (F-statistic):           9.08e-51
Time:                        22:37:27   Log-Likelihood:                0.48280
No. Observations:                 100   AIC:                             19.03
Df Residuals:                      90   BIC:                             45.09
Df Model:                          10                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
var_1          0.9817      0.028     35.336      0.000         0.927     1.037
var_2         -0.0011      0.025     -0.043      0.966        -0.052     0.050
var_3          0.0098      0.025      0.393      0.695        -0.040     0.059
var_4          0.0253      0.030      0.856      0.394        -0.033     0.084
var_5          0.0160      0.027      0.596      0.553        -0.037     0.069
var_6         -0.0138      0.028     -0.486      0.628        -0.070     0.043
var_7          0.0100      0.024      0.418      0.677        -0.037     0.057
var_8         -0.0358      0.027     -1.335      0.185        -0.089     0.017
var_9          0.0180      0.026      0.707      0.482        -0.033     0.069
var_10        -0.0574      0.025     -2.288      0.024        -0.107    -0.008
==============================================================================
Omnibus:                        5.760   Durbin-Watson:                   1.815
Prob(Omnibus):                  0.056   Jarque-Bera (JB):                2.903
Skew:                          -0.147   Prob(JB):                        0.234
Kurtosis:                       2.219   Cond. No.                         1.72
==============================================================================

This is one of the reasons why $R^2$ is rarely used when doing model selection.

Answer 2

There are a lot of things going on here, probably more than can be addressed in this forum. I will mention some of the major concerns, but you really should consult a regression textbook, class, or statistical consultant.

Due to missingness you are using less that a third of your observations. That should cause concern. Is the data missing at random? completely at random? or could the missingness be related to the variables in the model in an informative way? (if the last, then everything is suspect, if one of the former then you should really look at methods like multiple imputation or others to deal with missing values).

Each of the p-values is measuring the effect of that variable given that the other variables are in the model. It is possible that some of the "non-significant" variables are very important predictors, only that they are redundant given other (possibly also "non-significant") variables in the model. If you remove all the non-significant terms then it is possible that you would be removing some very important predictors.

It is not clear what question you are really trying to answer, and what to do next really depends on the question to be answered. If you are just trying to find a model for prediction, then something like "lasso" may be the next step (including cross-validation of your predictions). If you are interested if certain predictors are important given other predictors in the model, then a full-reduced test is appropriate. Other questions lead to other next steps, but to really understand it requires study and understanding, not just "this guy on the internet said ...".

Answer 3

Your regression analysis is suffering from 'data overload/clutter'.

First: You want to make sure your data samples were collected from a normalized distribution (meaning, you want each independent variable's data samples to be greater than 25 items. Anything less starts introducing data biasness and will form inaccurate conclusions). So get rid of the independent variables that you do not have 25 or more data samples from. And if you must use non-normal distributions you should use the Box-Cox procedure to convert and "normalize" them.

Second: You want to use data where there's some paired observations. That will help reduce 'accidental correlations'. For example, a fast food restaurant may notice that high customer satisfaction surveys seem to indicate preference for lower prices and bigger portion sizes. So you would have to collect surveys that have high customer satisfaction and see what the portion size was and what the price was that day and use those data sets. We are not using random variables like population density with no observable correlation.

Next, I would run each independent variable separately as a simple linear regression. The Rsquare is showing you the strength of that correlation, BUT YOU MUST VERIFY that this is not a chance variation. To do this, take a look at the calculation's P Value and compare it to the significant value you chose (usually .05, although you mentioned you are using .10). If the P Value is LESS than the significant value then the results are SIGNIFICANT, meaning not a chance variation. But that's not all, you also need to check the significance of the coefficient of the independent variable. Do this by finding the calculated 2-TAIL P VALUE next to your independent variable. If that 2-tail p value is LESS than your significant value, then the coefficient IS SIGNIFICANT. You should be able to see this visually as a scatter plot as well.

Once you do this for each independent variable you will be able to weed out the insignificant ones this way. Then run a multiple regression test for the strongest independent variables. Remember to check your 2-tail p values for each of the coefficients (independent variables). If they are less than the significance level then THEY ARE SIGNIFICANT and the correlation to the dependent variable is not by chance, it is true. It is not unusual to have some coefficients calculate to be significant, and others not. Remember this is showing CORRELATION not CAUSALITY.

Really the goal is to verify or analyze those independent variables with the strongest correlation to the dependent variable so you can get a working equation to predict how changes (increases or decreases) in these independent variables might affect the dependent variable. I would try to narrow it down to 3 independent variables.

http://www.wessa.net/rwasp_multipleregression.wasp

When you run the multiple regression for 3 variables you'll see it gives you an estimated Regression equation. Sales (Y) = -2.95 + 0.0149(portionsize) + .5572(price)

Simply multiply in values for the (portionsize) and (price) to see how they affect sales.

For your research, you need to reduce your data down to the important variables that are statistically significant; which independent variables do I want to know (realistically) if I could increase, or decrease, how will this impact my dependent variable.