5
$\begingroup$

I have been working on a problem whose goal is to find the most accurate coefficient (from a physical point of view) that expresses the relationship between a feature and an output variable. The model is linear, has multiple features, and I am using the normal equation to solve this.

My concern here is that some features are highly correlated, and in my understanding this can lead to some questions about how to find the correct coefficients. I have read that removing these features is sometimes recommended, but in this problem these features are not redundant, i.e. they all contribute to explaining the output variable.

So, how should I proceed to find the correct coefficients without discarding any features?

Edit: I think I should have provided a simple example of the real problem to make this more clear and interesting:

  • Consider a small village where electricity consumers are connected to the same grid.
  • I would like to know how the voltage at my home is affected by the consumption of consumer A, consumer B, ....
  • Part of these consumers have PV panels, which inject power into the grid (it's like having a negative consumption).
  • Injecting power affects the voltages of the grid, particularly near that injection point. For example, if my neighbor is injecting power into the grid, the voltage at my home will increase more than if that injection occurred farther.
  • During the day, electricity consumption may be low because people are not at home but all the PV panels are injecting power in a very similar way. As you can see bellow, when consumers install PV panels, their consumption patterns become highly correlated. However, I cannot discard any of them because each one is influencing my voltage. Closer to me, higher the influence.

enter image description here enter image description here

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ What have you heard about correlated variables leading to questions about estimating the coefficients? $\endgroup$
    – Dave
    Feb 22, 2022 at 13:12
  • $\begingroup$ @Dave: From what I understood, if two variables are highly correlated it may be difficult to extract the real influence of each one of these feature on the output. For prediction it may not be a problem but in my case that influence has a physical meaning that I need for other purposes. $\endgroup$
    – user10279396
    Feb 22, 2022 at 13:36
  • $\begingroup$ Correlated variables means that once you know about one, you know about the other. It would be similar to including a variable twice in the model. You have to either use one or the other or a combination of both such as a sum or a ratio. $\endgroup$
    – LDBerriz
    Feb 22, 2022 at 17:56
  • 3
    $\begingroup$ @LDBerriz That is false. Please see my answer here. $\endgroup$
    – Dave
    Feb 22, 2022 at 18:21
  • $\begingroup$ Looks that your model should be [Voltage in my home] ~ Cons[A,B,...] + PV. $\endgroup$
    – LDBerriz
    Feb 23, 2022 at 19:21

1 Answer 1

Reset to default
10
$\begingroup$

The correlation affects the standard errors on coefficients, but, at least under nice conditions that are typical to assume, the OLS solution is unbiased. Regress away, knowing that there is uncertainty in your estimates. If that uncertainty is small enough (that's a decision for you to make with your knowledge of the material being studied), then you're done. If you are concerned about lacking the sample size to have small standard errors that result in less uncertainty, then you can do a sample size calculation to determine how many observations you need to get acceptable results. An alternative could be to design your experiment to keep the features from getting correlated. How to execute such an experiment will come down to your knowledge of the subject matter.

Alternatives like regularized regression might be problematic in your case where interpreting the coefficients is of paramount importance. In particular, there is not a straightforward way to estimate uncertainty in the coefficient estimates, especially if you use cross validation to tune the regularization amount.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Note the importance of knowing the acceptable uncertainty before you do the statistics. If you run the regression, notice that the confidence intervals are too wide, and want to collect more data, you're in dangerous territory. $\endgroup$
    – Dave
    Feb 22, 2022 at 18:34
  • $\begingroup$ The choice is between (1) OLS regression with relatively inaccurate (because of high variance, no bias, ultimately high mean squared error) point estimates with known uncertainty and (2) ridge regression with relatively accurate (because of low variance, some bias, ultimately lower mean squared error) point estimates with unknown uncertainty. If the OP is after accuracy, the latter choice seems more directly relevant than the former. That kind of goes against your approach. Yet another alternative: a Bayesian counterpart to ridge regression that has both high accuracy and known uncertainty. $\endgroup$
    – Richard Hardy
    Feb 23, 2022 at 7:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.