Return to Question

Okay, this is a thought experiment:

Suppose you have a dataset with 40 covariates. Suppose the data has every nice theoretical property you could want: randomly sampled, variables not correlated, etc. Suppose it is also the case that the null is true for each beta coefficient at a 95% level AND we don't know that the null is true in all cases.

Next, you run the model, and 2 of the 40 betas appears to be significant, with a modestly large t value, but nothing crazy. Certainly enough to reject the null for both of these betas. Now, given that it was assumed the null is true, this is a perfectly reasonable result. Each beta has an approximately 5% chance of rejecting, so you would have expected two betas to reject before you even run the model!

This brings me to my issue: in what sense can we say multiple betas are significant when we ALWAYS expect a certain number to reject just due to randomness? AND how is it possible to conduct hypothesis tests for these betas simultaneously? Isn't that cheating, sort of like rolling a die until you get a six? It seems to me you can only pre-specify a single beta to look at before you run the model to get "true" significance.

To clarify what I mean by simultaneous: I am not talking about the overall test (F-Test, for example) or any related mitigating actions like multiple comparison corrections. I fully agree that these tests and measures are effective and make sense. I am asking at the level of the individual betas themselves with their individual CIs: if two betas reject, should I just ignore this and say: "Well, I expected two to reject. This effects are not real."? This may be my main point: does significance at the regression coefficient level mean anything in this case? Or suppose four had rejected. I would guess two would be false signals by chance, but how do I know which is real and which is fake?

PS: If you have an open-source text to cite and support your answer, I would love that.

Okay, this is a thought experiment:

Suppose you have a dataset with 40 covariates. Suppose the data has every nice theoretical property you could want: randomly sampled, variables not correlated, etc. Suppose it is also the case that the null is true for each beta coefficient at a 95% level AND we don't know that the null is true in all cases.

Next, you run the model, and 2 of the 40 betas appears to be significant, with a modestly large t value, but nothing crazy. Certainly enough to reject the null for both of these betas. Now, given that it was assumed the null is true, this is a perfectly reasonable result. Each beta has an approximately 5% chance of rejecting, so you would have expected two betas to reject before you even run the model!

This brings me to my issue: in what sense can we say multiple betas are significant when we ALWAYS expect a certain number to reject just due to randomness? AND how is it possible to conduct hypothesis tests for these betas simultaneously? Isn't that cheating, sort of like rolling a die until you get a six? It seems to me you can only pre-specify a single beta to look at before you run the model to get "true" significance.

To clarify what I mean by simultaneous: I am not talking about the overall test (F-Test, for example) or any related mitigating actions like multiple comparison corrections. I fully agree that these tests and measures are effective and make sense. I am asking at the level of the individual betas themselves with their individual CIs: if two betas reject, should I just ignore this and say: "Well, I expected two to reject. These effects are not real"? This may be my main point: does significance at the regression coefficient level mean anything in this case? Or suppose four had rejected. I would guess two would be false signals by chance, but how do I know which is real and which is fake?

PS: If you have an open-source text to cite and support your answer, I would love that.

Okay, this is a thought experiment:

Suppose you have a dataset with 40 covariates. Suppose the data has every nice theoretical property you could want: randomly sampled, variables not correlated, etc. Suppose it is also the case that the null is true for each beta coefficient at a 95% level AND we don't know that the null is true in all cases.

Next, you run the model, and 2 of the 40 betas appears to be significant, with a modestly large t value, but nothing crazy. Certainly enough to reject the null for both of these betas. Now, given that it was assumed the null is true, this is a perfectly reasonable result. Each beta has an approximately 5% chance of rejecting, so you would have expected two betas to reject before you even run the model!

This brings me to my issue: in what sense can we say multiple betas are significant when we ALWAYS expect a certain number to reject just due to randomness? AND how is it possible to conduct hypothesis tests for these betas simultaneously? Isn't that cheating, sort of like rolling a die until you get a six? It seems to me you can only pre-specify a single beta to look at before you run the model to get "true" significance.

To clarify what I mean by simultaneous: I am not talking about the overall test (F-Test, for example) or any related mitigating actions like multiple comparison corrections. I fully agree that these tests and measures are effective and make sense. I am asking at the level of the individual betas themselves with their individual CIs: if two betas reject, should I just ignore this and say: "Well, I expected two to reject. This effects are not real."? This may be my main point: does significance at the regression coefficient level mean anything in this case?

PS: If you have an open-source text to cite and support your answer, I would love that.

Okay, this is a thought experiment:

Suppose you have a dataset with 40 covariates. Suppose the data has every nice theoretical property you could want: randomly sampled, variables not correlated, etc. Suppose it is also the case that the null is true for each beta coefficient at a 95% level AND we don't know that the null is true in all cases.

Next, you run the model, and 2 of the 40 betas appears to be significant, with a modestly large t value, but nothing crazy. Certainly enough to reject the null for both of these betas. Now, given that it was assumed the null is true, this is a perfectly reasonable result. Each beta has an approximately 5% chance of rejecting, so you would have expected two betas to reject before you even run the model!

This brings me to my issue: in what sense can we say multiple betas are significant when we ALWAYS expect a certain number to reject just due to randomness? AND how is it possible to conduct hypothesis tests for these betas simultaneously? Isn't that cheating, sort of like rolling a die until you get a six? It seems to me you can only pre-specify a single beta to look at before you run the model to get "true" significance.

To clarify what I mean by simultaneous: I am not talking about the overall test (F-Test, for example) or any related mitigating actions like multiple comparison corrections. I fully agree that these tests and measures are effective and make sense. I am asking at the level of the individual betas themselves with their individual CIs: if two betas reject, should I just ignore this and say: "Well, I expected two to reject. This effects are not real."? This may be my main point: does significance at the regression coefficient level mean anything in this case? Or suppose four had rejected. I would guess two would be false signals by chance, but how do I know which is real and which is fake?

PS: If you have an open-source text to cite and support your answer, I would love that.

Okay, this is a thought experiment:

Suppose you have a dataset with 40 covariates. Suppose the data has every nice theoretical property you could want: randomly sampled, variables not correlated, etc. Suppose it is also the case that the null is true for each beta coefficient at a 95% level AND we don't know that the null is true in all cases.

Next, you run the model, and 2 of the 40 betas appears to be significant, with a modestly large t value, but nothing crazy. Certainly enough to reject the null for both of these betas. Now, given that it was assumed the null is true, this is a perfectly reasonable result. Each beta has an approximately 5% chance of rejecting, so you would have expected two betas to reject before you even run the model!

This brings me to my issue: in what sense can we say multiple betas are significant when we ALWAYS expect a certain number to reject just due to randomness? AND how is it possible to conduct hypothesis tests for these betas simultaneously? Isn't that cheating, sort of like rolling a die until you get a six? It seems to me you can only pre-specify a single beta to look at before you run the model to get "true" significance.

To clarify what I mean by simultaneous: Note that I am not asking about how we can test if at least one regression coefficient is significant (which is another way of saying test all regression coefficients at once and reject if at least one is useful), which we can address with an F-test. I am also not asking how to control the overall type I error rate to be 95%, which can be done with Tukey, etc. I'm asking, given you are only looking at the statistically significant betas (in this thought experiment, the two betas that rejected), why do we have any reason to believe they are both actually significant? One could actually predict two would reject if they assume the null is true.

PS: If you have an open-source text to cite and support your answer, I would love that.

Okay, this is a thought experiment:

Suppose you have a dataset with 40 covariates. Suppose the data has every nice theoretical property you could want: randomly sampled, variables not correlated, etc. Suppose it is also the case that the null is true for each beta coefficient at a 95% level AND we don't know that the null is true in all cases.

Next, you run the model, and 2 of the 40 betas appears to be significant, with a modestly large t value, but nothing crazy. Certainly enough to reject the null for both of these betas. Now, given that it was assumed the null is true, this is a perfectly reasonable result. Each beta has an approximately 5% chance of rejecting, so you would have expected two betas to reject before you even run the model!

This brings me to my issue: in what sense can we say multiple betas are significant when we ALWAYS expect a certain number to reject just due to randomness? AND how is it possible to conduct hypothesis tests for these betas simultaneously? Isn't that cheating, sort of like rolling a die until you get a six? It seems to me you can only pre-specify a single beta to look at before you run the model to get "true" significance.

To clarify what I mean by simultaneous: I am not talking about the overall test (F-Test, for example) or any related mitigating actions like multiple comparison corrections. I fully agree that these tests and measures are effective and make sense. I am asking at the level of the individual betas themselves with their individual CIs: if two betas reject, should I just ignore this and say: "Well, I expected two to reject. This effects are not real."? This may be my main point: does significance at the regression coefficient level mean anything in this case?

PS: If you have an open-source text to cite and support your answer, I would love that.