1
$\begingroup$

I ran a regression $Y \sim X_1 + X_2 + .. X_n$. I find out what one regressor , $X_1$'s performance (or correlation with $Y$) depends on another variable $t$ (not in the regression). So basically if I bucket by $t$, within each bucket of $t$, I plot the correlation of $X_1$ and $Y$. I can see that when $t$ is higher, the correlation between $X_1$ and $Y$ is very obviously higher.

One example is $t$ is time of the day. $X_1$ has higher correlation with $Y$ as time goes by during the day, for may reason. But $t$ itself doesn't predict $Y$ at all. My $t$ and $X_1$ can be positive or negative.

So I want to take advantage of this phenomenon. I scale $X_1' = X_1 * (1+t)$. Basically when $t>0$, I increase the magnitude of $X_1$, and vice versa. I rescale $t$ to be within $(0,1)$ or even smaller intervals. Then I run regression $Y \sim X_1' + X_2 + .. X_n$

Much to my surprise, this adjustment doesn't improve my regression at all, based on all the regression stats.

How do I improve my regression based on this observation?

Improve this question
$\endgroup$
4
  • $\begingroup$ What do you mean by $X_1$'s performance? $\endgroup$
    – Sextus Empiricus
    Apr 18, 2022 at 18:28
  • $\begingroup$ Which regression stats do you use to assess the improvement of the regression? $\endgroup$
    – Sextus Empiricus
    Apr 18, 2022 at 18:29
  • $\begingroup$ I meant its correlation with Y. The improvement of the regression is accessed by the typical metrics like R^2, correlation of prediction with final results, etc. $\endgroup$
    – Matt Frank
    Apr 18, 2022 at 18:38
  • $\begingroup$ Stronger correlation does not mean that the coefficient needs to be larger. However, by multiplying $x_1$ with $1+t$ this is what you effectively do. $\endgroup$
    – Sextus Empiricus
    Apr 18, 2022 at 19:12

1 Answer 1

Reset to default
2
+50
$\begingroup$

When you are multiplying $X_1$ and $t$, then you are using something like an interaction effect.

You could apply such an effect in your regression, but based on the correlation alone it is difficult to judge whether you have exactly an interaction effect, where the magnitude of the coefficient associated with $X_1$ varies for different values of $t$. Instead, the situation could also be that the coefficient remains the same for different values of $t$, but you have a decreasing correlation because the noise or other parameters change.

What are the coefficient values for the different buckets? Could you plot these? Also, even if we would see a variation in the coefficient as function of the bucket $t$, then it might be that you need to apply it to multiple variables at the same time.

If correlation of $X_1$ and $Y$ are close to zero, Shoudln't the coefficient be close to zero, thus smaller? And how else can I leverage this feature?

The coefficients size is not the same as correlation $\rho$. In simple linear regression it is for instance $\beta = \rho \frac{\text{var}Y }{\text{var} X_1}$. And for the more general ordinary linear regression there are even more factors involved.

Thanks. Then what else can I do to "condition" $X_1$ to enhance my regression? Seems a bit wasteful not to use this phenomenon?

It is not necessarily possible that you could 'enhance' the regression. It might be that you just have a fixed deterministic part of the model (not dependent on $t$) but noise that is not constant in time. In that case you could 'enhance' the regression by using weighted regression giving points with different $t$ different weights, but it is not gonna show much in the regression statistics like lower $R^2$. In fact, the $R^2$ will get worse because you focus on the part of the data with lower variance.

Below is an example with the model

$$z = a + b \sin(15 \pi t) + \text{noise}(t)$$

where the correlation changes depending on the time frame. For $t>10$ the correlation is lower, but the coefficient remains the same. The change in correlation is in this example due to the different noise.

When we want to make use of this 'phenomenon' then we could use for instance a weighted least squares model. This will reduce the impact of the noisy points on the estimate. This is useful because the fit with the weighted least squares is closer to the 'true' model. However, the red line based on a regular least squares fit, results in a higher $R^2$ (because it fits the noise). You might consider that a better performance, but it is deceptive and it is just a better fit of the noise and not a better fit of the underlying model.

example of better fit with worse R^2

set.seed(1)
x = seq(0,1,0.01) * 15 * pi
sig = c(rep(0.3,floor(length (x)/3)),
        rep(2,length(x) - floor(length (x)/3)))
noise = rnorm(length(x), 0, sig)
y = sin(x)
z = y + noise
plot(x,z, pch=21, col=1, bg=1,cex=0.7)
lines(x,y)

mod = lm(z~y)
lines(x,predict(mod), col = 2)

mod2 = lm(z~y, weights = 1/sig^2)
lines(x,predict(mod2), col = 3)

legend(0,5, c("true model", "least squares fit", "weighted least squares fit"), cex = 0.7, lty = 1, col = c(1,2,3))
Improve this answer
$\endgroup$
6
  • $\begingroup$ Indeed. the coefficient size doesn't seem to correlate with the bucket of t. Struggle to understand why though. If correlation of $X_1$ and $Y$ are close to zero, Shoudln't the coefficient be close to zero, thus smaller? And how else can I leverage this feature? $\endgroup$
    – Matt Frank
    Apr 18, 2022 at 19:48
  • 1
    $\begingroup$ @MattFrank the coefficients size is not the same as correlation $\rho$. In simple linear regression it is for instance $\beta = \rho \frac{\text{var} Y}{\text{var} X_1}$. For the more general ordinary linear regression there are even more factors involved. $\endgroup$
    – Sextus Empiricus
    Apr 18, 2022 at 19:51
  • $\begingroup$ Thanks. Then what else can I do to "condition" $X_1$ to enhance my regression? Seems a bit wasteful not to use this phenomenon? $\endgroup$
    – Matt Frank
    Apr 18, 2022 at 19:55
  • $\begingroup$ It is not necessarily possible that you could 'enhance' the regression. It might be that you just have a fixed deterministic part of the model (not dependent on $t$) but noise that is not constant in time. In that case you could 'enhance' the regression by using weighted regression giving points with different $t$ different weights, but it is not gonna show much in the regression statistics like lower $R^2$. In fact, the $R^2$ will get worse because you focus on the part of the data with lower variance. $\endgroup$
    – Sextus Empiricus
    Apr 18, 2022 at 20:00
  • $\begingroup$ @SextusEmpiricus the Wikipedia article you reference states that $\beta = \rho \frac{\text{sd}Y}{\text{sd}X_1}$ , rather than the ratio of variances factor you mention. $\endgroup$
    – Kuku
    Apr 20, 2022 at 8:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.