4
$\begingroup$

I had a discussion with a colleague about a research project he is involved in.

We're living in a regression framework, where the colleague aims to infer about the causal effect of an independent variable X on a dependent variable Y. A model is assumed:

$Y_i = \beta_0 + \beta_1X_i + ... +\epsilon_i$

where the typical assumptions are made about the distributions and the error etc.

Now, the colleague's 'personal' research hypothesis is that the true coefficient $\beta_1$ should be very small or even 0. Meaning: X has little or no effect on Y.

To learn about his hypothesis, he estimates the model parameters (coefficients) by using least squares, obtaining an estimate $\hat\beta_1$.

After obtaining the estimate a hypothesis test is conducted:

$H_0: \beta_1 = 0$ and the p-value of the test statistic is observed.

This is the standard test which is displayed after conducting simple ols, by using ordinary programs like R or Stata.

Case 1:
Now, my colleague argues that observing a very low estimate for the coefficient e.g $\hat\beta_1$ = 0.1 in combination with a very low p-value e. g. p=0.001 (highly significant) would be strong evidence for his personal hypothesis of the true $\beta_1$ being very small or 0.

Case 2:
He also says, that the Case 1 example values would be stronger evidence in favor of his hypothesis compared with an example where the coefficient estimate would be some higher value e.g. $\hat\beta_1$ = 0.6 and the p-value would be e.g. p=0.95 (highly insignificant).

To break the argument down in one sentence: The claim is the case 1 values offer stronger evidence in favor of the 'personal' hypothesis that the true $\beta_1$ is very small or 0 compared with the values of case 2; meaning, X has little or no effect on Y.

Now I understand the argument behind the point but I don't really agree on this because in case 1 we reject the hypothesis that the true coefficient is 0, whereas in case 2 we cannot reject the hypothesis that it is actually 0.

Naturally, the question is:

Which case of values should a researcher desire who wants to obtain evidence in favor of their personal hypothesis of the true coefficient being very small or 0? (The personal hypothesis is that X has little or no effect on Y.)

Note: On purpose, I leave the expression 'very small or 0' as vague as it is. I also accept answers which demand that it cannot be answered without making this statement more explicit. Hoping for a discussion where the statement is as vague as it is though, because I often observe this in applied empirical discussions.

Improve this question
$\endgroup$
14
  • 1
    $\begingroup$ There is nothing -like a personal hypothesis . Assuming that we need to decide between two situations, interpretation of results can be made. $\endgroup$
    – user10619
    Dec 10, 2017 at 11:04
  • 2
    $\begingroup$ Have you thought of using confidence intervals? A narrow interval around a low value would suggest your colleague is justified. $\endgroup$
    – mdewey
    Dec 10, 2017 at 13:58
  • 1
    $\begingroup$ @subhash. Low significant coefficient is short for : "The estimated coefficient (beta hat) is small and has a resulting p-value below a desired threshold, e.g. p<0.05". Low true coefficient is short for (assuming the specified model is correct) : "The true value of the parameter beta, which we can never observe but we try to estimate with beta hat". I totally agree with the second part of your statement, iam not sure if this makes the question meaningless though. $\endgroup$
    – kirtap
    Dec 10, 2017 at 14:22
  • 2
    $\begingroup$ I agree with @Mdewey that confidence intervals are a way forward. It's important not to be distracted by details here. Strength and significance of effect are different properties, not reducible to a common scale. It is like asking for a basket-ball player who is tall for vertical challenges and fast for horizontal, and then saying How do I choose between a tall slower player and a fast shorter player? $\endgroup$
    – Nick Cox
    Dec 10, 2017 at 14:33
  • 1
    $\begingroup$ @kirtap "Nevertheless, in general a common practice is to only interpret the coefficient if a desired p-value for some test is reached." that is absolutely and summarily false. $\endgroup$
    – AdamO
    Apr 24, 2018 at 21:26

2 Answers 2

Reset to default
4
$\begingroup$

Really your investigator needs to reframe the problem. As stated, if they wish to show (via significance testing) there is little or no effect, they must assume the contrary. Null hypothesis significance testing controls the type one error. The type two error cannot be controlled (in a two tailed test) because the "truth" can be arbitrarily close to the null.

However a well designed study with high power and high precision is often treated as a confirmatory study, where either null or significant findings are treated as evidence for or against an effect.

To show that findings are consistent with a range of zero to small (non clinically significant) effects, you must conduct a noninferiority test which assumes a margin of effect. Choosing the margin is discussed elsewhere in the literature. The margin value is not a statistical problem, it is chosen based on the scientific understanding of the question.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Here clinically means substantively, physically, economically, whatever, as the case may be. $\endgroup$
    – Nick Cox
    Dec 10, 2017 at 14:54
  • $\begingroup$ This sounds very understandable. Could you please give some more details for your second paragraph and how it is connected to your first and third? $\endgroup$
    – kirtap
    Dec 12, 2017 at 23:05
  • $\begingroup$ @kirtap cases of well designed studies with high precision mean that we can consider approaches different than NHSTs. We can merely inspect the range of the CI and immediately apply the results to the population of interest. For instance, if I access breast cancer screening records on 100,000 women of all races/ethnicities and their breast cancer incidence, I can say something meaningful about a null result: it means that screening does not in fact reduce breast cancer risk. This is a controversial finding that continues to be a moot point among screening advocates. $\endgroup$
    – AdamO
    Jun 28, 2018 at 16:19
1
$\begingroup$

Sounds to me like the hypothesis is not $H_0: \beta_1=0$ but rather that you have $H_{0c}:|\beta_1|\leq c$ for some suitably chosen "small value" $c $. This value would likely be based on the scale of $X_1$.

You then have two situations. - $|\hat {\beta_1}|\leq c $...you cannot reject the null (p-value is $1$ because the mle is in the null hypothesis space) - $|\hat {\beta_1}|> c $...you might reject the null if the gap is big enough (as the data fits into the alternative hypothesis space)

The test could be done using likelihood ratio.

This justifies your friend's view, provided $c $ is larger than $0.1$ and smaller than $0.5$.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.