2
$\begingroup$

Let's say I estimate

$d_t = b + \epsilon_t$,

where $d$ stands for some difference between two variables and $b$ is a constant. I estimate this model $i$ times for $i$ different subjects (say firms, individuals, etc.). Put differently, I simply estimate the mean of the differences of two variables for $i$ subjects. Let's say $b$ turns out to be insignificant for every $i$.

I then estimate

$b_i = \gamma_0 + \gamma_1 x_i + \epsilon_t$,

where $b_i$ is the estimation of $b$ for every $i$, $\gamma_0$ is a constant and $\gamma_1$ is the coefficient of some predictor $x_i$.

Can, $\gamma_1$ be significant, eventhough the constants $b_i$ are insignicant themselves? If so, how would you interpret such a significant influence in a meaningful way?

$\endgroup$
2
+50
$\begingroup$

The example below might help to intuitively understand it. It shows a plot of datapoints $d$ (black dots) and the estimates $\hat{b}$ of the population means (blue squares) with the error bars relating to the standard error of the $\hat{b}$. Also shown is a (red) line indicating the linear model for the estimates $\hat{b}$ as a function of the $x$.

So we see that all those individual estimates have each not much accuracy and their difference from zero is not significant.

However because there are so many measurements for the different values of $x$ we can still see a reasonable certain relationship for the $\hat{b}$ as function of $x$.

In order to determine the significance of the linear relationship a lot more data is combined together. That is why you can get the significant relationship for the line b ~ x, but each of the individual points is not significant.

example illustration

This situation also occurs often when people compare two curves. Some researcher may have taken multiple measurements for each value $x$ and based on a pointwise overlap of error bars the conclusion might be that there is no difference. However, for a linear curve, or some other curve (which takes all the data into account together) the power of a test for differences is much higher. This is why I do not so often focus on making triplicate measurements. When you know the underlying model well then you do not need to take multiple measurements at every single vale of the independent variable $x$, that is because you are not comparing the single points but instead the estimates for the model coefficients.

Code for the graph

Steps:

  1. Use an independent variable $x$ with values $-10, -9, -8, \dots, 9, 10$
  2. Model unknown variable $b$ according to: $$b \sim N(0.01 x, 0.01^2)$$
  3. Model dependent variable $d$ according to $$d \sim N(b, 0.2^2)$$
  4. Compute estimates $\hat{b}$ (and determine their significance, which only turns out significant here for the point at x=-5, with p-value 0.006) and perform regression for $\hat{b}$ as function of $x$ (which turns out significant with p-value <0.001

--

set.seed(1)
ns <- 10

# create data
x <- seq(-10,10,1)
b <- rnorm(length(x),mean = 0.01*x,sd = 0.01)
d <- matrix(rep(b,ns),ns, byrow = 1)+rnorm(ns*length(x),0,0.2)
b_est <- colMeans(d)

# blank plot
plot(-100,-100, xlim = c(-10,10), ylim = c(-0.5,0.5), 
     xlab = "x", ylab = "d")

## model for b ~ x
mod <- lm(colMeans(d) ~ x)
summary(mod)
lines(x, predict(mod), col = 2)

# line for reference
lines(c(-20,20), c(0,0), lty = 2)

# add points
for (i in 1:length(x)) {
  # raw data 'd'
  points(rep(x[i],ns),d[,i],pch = 21, col = 1, bg = 1, cex = 0.4)

  # significance of 'b'
  mt <- t.test(d[,i])
  if (mt$p.value < 0.05) {
    text(x[i],0.5,"*",col=2)
  }

  # estimates 'b'
  mod <- lm(d[,i] ~ 1)
  points(x[i],mod$coefficients[1],
         pch = 22, col = 4, bg = 4)

  # error bars
  err <- summary(mod)$coef[2]
  mea <- summary(mod)$coef[1]
  arrows(x[i], mea+err, x[i], mea-err, length=0.05, angle=90, col=4, code = 3)
}

legend(-10,0.5, c("data points 'd'",
                  "estimates 'd ~ b'",
                  "relationship b ~ 1+x"),
       col = c(1,4,2), pt.bg =c(1,4,2),lty = c(NA,NA,1), pch = c(21,22,NA), pt.cex = c(0.4,1,1),
       cex = 0.7)
| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Excellent example. Thank you. $\endgroup$ – shenflow Jun 5 at 22:22
0
$\begingroup$

Yes this is very well possible. It means that the $b$'s have barely an effect on the $d$'s. But there's no reason that this makes that you can not model the $b$'s with some $x$'s.

  • This happens clearly when $b$ and $d$ are completely unrelated. E.g. say $d$ is the degree that somebody likes blue more than red, $b$ is somebody's cholesterol level, and $x$ is somebody's saturated fat consumption.

    It is not unimaginable that in some experiment/measurement $x$ (fat consumption) will have a significant effect on $b$ (cholesterol level) in a linear model. But $b$ is not significantly related with $d$ preference for colour.

  • The same thing happens also when $b$ and $d$ are likely related (ie. not like the earlier contrived example with clearly unrelated things). For instance $d$ could be some health outcome, like the risk of getting coronary hearth disease.

    In health research (and I am sure other fields have this as well) it happens quite often that some behaviorial parameter $x$ like (eating habits, exercise, etc.) has a significant (measurable) effect on some physiological parameter $b$ (like cholesterol level, body fat percentage, bone strength, etc), but the effect $b$ (and $x$ as well) on some health outcome $d$ (like risk of disease, disability or death) is not so clear and often does not turn into a significant effect in some measurement/experiment.


Some more particular (more extreme) case is when $x$ is found to strongly correlate with each of $b$ and $d$, but $b$ and $d$ do correlate a lot with each other.

This is possible. For given correlations among three variables $\sigma$, $\tau$, and $\rho$, there is a range of values that $\rho$ can take (within some bounds) depending on the other two correlations.

$$\sigma\tau - \sqrt{(1-\sigma^2)(1-\tau^2)} \le \rho \le \sigma\tau + \sqrt{(1-\sigma^2)(1-\tau^2)}$$

So this can make that for some data the effect of $x$ in a linear relation is significant for both $b$ and $d$, but still $b$ is not a significant effect in a linear relation for $d$. (In this case, when some weak correlation is present, then it might be the case that more measurements, a larger sample, will show a significant effect)

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ You treated $b$ like the coefficient of some variable. BUT it is just an intercept NOT the coefficient of some variable (e.g. cholestorol.). In my particular example - if all the $b$ are not-significantly different from 0, the regression of $x$ on $b$ is basically just a regression of some values ($x$) on a vector of zeroes (because each $b$ would more or less equal zero as the first model indicates). The only case where $\gamma_1$ (the coefficient of $x$) would then be significantly different from 0 is if $x$ would be relatively constant across all observations. This is my take on it.. $\endgroup$ – shenflow Jun 5 at 16:46
  • $\begingroup$ @shenflow maybe you should give a more practical example because now I am confused. What do you mean by "let's say $b$ turns out to be insignificant" what test are you doing in that case and what are your observations? $\endgroup$ – Sextus Empiricus Jun 5 at 17:08
  • $\begingroup$ Are $b$ estimates or observations? The way I see it now is that you observed d and x and b is an estimate for d which is not significantly different from 0 but x predicts significantly $b$. $\endgroup$ – Sextus Empiricus Jun 5 at 17:31
  • $\begingroup$ Empricius. $b$ is an estimate. It is the intercept of an intercept only model, i.e. the mean. If $d$ is 0 in its mean, $b$ is zero. Assume that this is the case in the "true population". However, empirically the values of $b$ are slightly different from 0 across all $i$ estimations. However it is not significantly different from 0 as t-tests indicate. Yet, I still want to exploit the fact that the actual estimated values are all slightly different from 0 (although t tests indicate that they are NOT significantly different from 0). More specifically, I want to regress $b$ on $x$. $\endgroup$ – shenflow Jun 5 at 22:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.