# Interpreting statistical significance

Suppose I repeatedly run the same regression model for a set of individuals and I am interested in determining whether a given independent variable has a statistically significant impact on the dependent variable.

To do so, I have computed the percentage of statistically significant coefficients at different significance levels based on their p-values, finding that about 15% to 20% of the coefficients are significant at a 10% level, 10% of the coefficients are significant at a 5% level and 3% to 4% of the coefficients are significant at a 1% significance level.

In the above scenario, would you say that there is evidence that the independent variable has a statistically significant effect on the dependent variable in the given sample? If yes, would the evidence be weak or relatively solid? And if not, why?

Please note that I am aware that alternative procedures such as panel regressions could be performed, but that is not the case for my dataset. I am specifically interested in the above circumstance.

• When you repeatedly run the same model, you should always get the same result. So you seem to by varying something between the runs. What is that? – Stephan Kolassa Sep 23 '19 at 9:32
• The model stays the same among regressions, what varies are the values taken by the dependent and independent variables. What I am doing is basically regressing the returns of a stock (dependent) on a set of market factors (independent), and then repeating the same process for each stock in the set. Consequently the dependent variable changes, and also the independent variables do because the period under consideration is not always the same. – Gianluca Sep 23 '19 at 9:53

Honestly, what you make is quite odd...but let us try to analyze it.

## Please note that I am happy if someone more expert can validate (or debunk or complement) my reasoning.

To simplify the situation, let us imagine that there is no missing data and that the time period under consideration is always the same (as I imagine it is quite overlapping).

You perform the same analysis (ie. the same regression model and the same independent variables) on q different samples (for $$p$$ different kinds of stock values: $$Y_1$$ for company 1,..., $$Y_j$$ for company j, ..., $$Y_q$$ company q).

In those case, let us consider the independent variable under investigation, for instance $$X_1$$. And let us note $$p_j$$, where $$j=1..q$$ the p-value associated with $$X_1$$. What is the interpretation of $$p_j$$? That, in case $$X_1$$ is truly uncorrelated with the $$Y_j$$, one would find a such a "high" value of $$\beta_1^j$$ in only $$p_j$$% of the cases.

Hence, that a given $$p_j$$ is over 10% or under 10% is meaningful. But the meaning of computing how many are over and under 10% is more difficult to gasp.

Maybe, let us imagine that these $$q$$ different samples are independent draws (a bold assumption considering stock values of various companies) of a same generative process of the form : your repeated regression + firm fixed-effects. In this case, $$\beta_1^1$$,..., $$\beta_1^j$$, ..., $$\beta_1^p$$ seem to be independent draws around the true value $$\beta_1$$ of the data generating process. Let us imagine you have 2,000 or even 40,000 individuals. A first idea would be to draw the distribution around $$\beta_1$$, and to check whether its mean is significantly different from zero. But this would dismiss all the variance information at the regression level. I would guess that comparing the p-values $$p_j$$ is similar to considering that all the variances of the $$\beta_j$$ are approximately the same (or that: $$V(\hat{V(\beta_1)})<). Hence, you get a distribution of p-values that has the same meaning as the distribution of the $$\beta_1$$, except that now you have to compare it with the distribution of the p-values under the null hypothesis, what you do above on a few points. Both distribution are probably significantly different, and your conclusion would be valid.

But that this reasoning is valid is conditional on some assumptions above being respected. I do not know how one could manage the (implausible) independence assumption. Hence, I would (personally and based only on the above considerations) not consider the evidence you present to be very solid.