9
$\begingroup$

I'm a beginner in stats, any help would be appreciated.

$\endgroup$
  • 1
    $\begingroup$ Correlation between parameters is not how you normally determine how good a model is. Without a measure of good or bad this question becomes very broad/subjective. (There is certainly something about correlation between parameters but the question her is posed in an indirect and very open way) $\endgroup$ – Sextus Empiricus Mar 27 at 14:12
  • $\begingroup$ Without clarification, we should take it OP means parameters as actual parameters of the model. In linear regression it's impossible to get uncorrelated parameters IIRC, increasing the estimate of the intercept necessarily means changing the estimate of the slope and vice-versa. In more complex models it becomes really hard to argue in terms of correlations between parameters. $\endgroup$ – Firebug Mar 27 at 14:34
  • $\begingroup$ What do you mean by "Better"? $\endgroup$ – Peter Flom Mar 28 at 12:49
11
$\begingroup$

This depends on what you mean by "make the model better". Do you want to use this model to say something about how the world works, or to make predictions?

  • if the covariates are uncorrelated, then the beta values associated with them will generally be close to independent. (This is related but not identical to the idea of parameter orthogonality.) This is useful if you want to interpret the betas as saying something about the real world and you don't want them to be confounded with each other.
  • if you are concerned about the accuracy of the model's predictions then it doesn't really make any difference. The beta values will be correlated, but the predictions will be unaffected. You could orthogonalise your covariates and that would completely change the definition and interpretation of beta, but the fitted values, residuals and predictions would be the same as before.
| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Your answer definitely gets to the heart of the problem, but I think it could be stated more succinctly. The interpretation of the a coefficient in regression analysis (a beta) is that it represents the mean change in the dependent variable for a unit change in the independent variable when all other independent variables are held constant. If any independent variables are correlated, the interpretability of your model/coefficients goes for a toss. $\endgroup$ – shinvu Mar 27 at 15:29
  • $\begingroup$ The above comment wasn't exactly my words, I picked most of it up from this wonderful article: Picked up from here: statisticsbyjim.com/regression/… $\endgroup$ – shinvu Mar 27 at 15:33
  • $\begingroup$ @shinvu, that's not necessarily true — it will depend on whether the covariates can be manually manipulated (and whether they actually were). This is essentially the difference between modelling p(Y|X) and P(Y|do(x)), to use Pearl's notation. But that's quite a digression in the context of this question, I think. $\endgroup$ – JDL Mar 27 at 15:33
  • $\begingroup$ What if the covariates in the true data generating process are highly correlated with each other? $\endgroup$ – TrynnaDoStat Mar 27 at 19:00
  • $\begingroup$ @trynnaDoStat, it still depends what you want to use the model for. If you are making predictions only with it then it doesn't much matter. If you are 'interpreting the betas' then the betas themselves will be (anti)correlated (and will have wider than expected s.e.), but that may be acceptable (well, it will have to be — there is nothing you can do while keeping the interpretation of beta the same, since if you orthogonalise then beta means a different thing now) $\endgroup$ – JDL Mar 30 at 7:53
6
$\begingroup$

I presume by parameters you mean the features, which is quite unusual as @whuber commented. The next paragraph follows on this assumption.

Not necessarily. Highly correlated features can cause multi-collinearity but this doesn't mean that a model with correlated features is worse than uncorrelated features. A model can have a set of correlated features that describe the target variable very well, or a set of uncorrelated features and that is not related to the target variable in any way.

For parameter estimate uncorrelatedness, using a similar idea, assume you have uncorrelated random features that are also not related to the target variable. Since features are totally random, the parameter estimates will also be and show no correlation. So, still hard to say the model is better if you have no correlation.

| cite | improve this answer | |
$\endgroup$
  • 5
    $\begingroup$ It would be helpful to indicate how you are interpreting this question, because "parameters are uncorrelated" can mean at least three very different things, ranging from (1) a Bayesian prior with correlation to (2) correlation of parameter estimates to (3) correlation of the variables. Your interpretation of "parameter" as meaning "feature" is unusual but possibly is what the OP meant. $\endgroup$ – whuber Mar 26 at 21:30
  • 3
    $\begingroup$ Definitely @whuber! I also wouldn't use 'parameter' to denote the features but thought that I kind of understood what OP meant, but w/o explaining it. $\endgroup$ – gunes Mar 26 at 21:34
  • 1
    $\begingroup$ I’d be shocked if the OP meant anything other than correlated features, even though “parameter” is an unorthodox way to say that. $\endgroup$ – Dave Mar 26 at 21:38
  • 1
    $\begingroup$ @MichaelSidoroff if the model is not linear then correlation of the coefficients is more general. (In addition couldn't there be practical cases that it is not a lot the same? The correlation of parameters relate to the inverse of the matrix $(X^tX)^{-1}$. I have not an intuitive view of that but, there is a difference, and I can imagine that under particular circumstances the situation is a bit more different) $\endgroup$ – Sextus Empiricus Mar 27 at 14:17
  • 1
    $\begingroup$ @Sextus Empiricus - I totally agree with you on the non-linear case. Here however, we are dealing with a linear model, and that's why I claimed that correlation in features and correlation yields correlation in coefficients. $\endgroup$ – Michael Sidoroff Mar 27 at 14:36
2
$\begingroup$

I agree with @gunes that you might stumble on cases that training on highly correlated features will yield better results than on an uncorrelated featureset, but provided that your features are good (i.e. explain the target well).

In my experience though, it's better to get rid of highly correlated features, because this will simplify your model, and won't harm the predictability too much (because if cor(x, y) is high - its enough to know either of those features to get the prediction).

For example if you have square feet of the house and number of rooms in it, those features are most likely are highly correlated, so you might consider taking just the most informative of them and by this simplifying the model, and still retain the accuracy.

On the other hand, if all your features are uncorrelated, each one of them gives your model a different perspective on the problem, which will help it generalize better.

Hope that helps. Cheers.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

I am not a statistician, so I would be happy to be corrected by the other users if this answer is wrong/naive. Anyway: from the point of view of a numerical analyst, I would say yes, it is better, because then you can conclude that the matrix to (pseudo-)invert is well conditioned, and hence your solution will not be highly sensitive to perturbations of the input data (i.e., the observations that you are trying to fit).

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

In my estimation, your question is more aligned with @whuber's third interpretation noted in the comments.

Here's a simple linear regression model:

$$ Y = \beta_{0} + \beta_{1}X_{1} + \epsilon. $$

I will assume you have already built a model and you are investigating the impact of a variable $X_{1}$ that you believe to have a causal effect on your dependent variable $Y$. At this point, you may want to investigate the effect of other variables on your outcome. However, you discovered that other features in your dataset are related to $Y$, or may predict $Y$, but have no association with $X_{1}$. In this case, I would argue that these variables can be safely omitted from your analysis. For the sake of this explanation, I assume you are not automating your choice of predictor variables and a basic explanatory model has already been considered.

One of the primary goals of regression analysis is to 'separate out' the association of $X_{1}$ with other variables on the right-hand side of the equation so we can examine $X_{1}$'s unique influence on $Y$. Now, here's a second model with a control variable, $X_{2}$, included:

$$ Y = \beta_{0} + \beta_{1}X_{1} + \beta_{1}X_{2} + \epsilon. $$

In general, two conditions must be met. First, the variable $X_{2}$ should also be associated with $Y$. Second, the variable should be correlated with $X_{1}$, but not perfectly correlated. If $X_{2}$ is correlated with $X_{1}$, then including it in the foregoing equation affords us the ability to examine the effect of $X_{1}$ on $Y$ while holding $X_{2}$ fixed. If, however, the latter condition is not met and $X_{2}$ is uncorrelated with $X_{1}$, then this variable can be dropped from the analysis. I would argue that it more likely should be dropped in cases where $X_{2}$ is explicitly measured, and explicitly included—and it is unrelated to the main explanatory variable(s) already in the model. Again, one important feature of multiple regression is to purge $X_{1}$'s correlation with $X_{2}$. Throwing in a series of orthogonal regressors, if large, decreases the precision of the estimated coefficients. So from my perspective, I wouldn’t say a model is “better” with more irrelevant controls on the right-hand side of your equation.

I agree with @MichaelSidoroff's answer that once a set of uncorrelated features enters the model and you didn't have any a priori theoretical basis for including them, then each factor offers a different perspective on the phenomenon under study. Note why multiple regression is often not necessary in most randomized studies. Randomization expels any correlation between the main treatment variable (independent variable) under study and other observed (and unobserved) characteristics of individuals. Thus, there is no need to explicitly control for the other observed factors across individuals using a multiple regression framework, because the correlation has been removed (or at least we hope it has).

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

It is a very good question. The concept related to your question is Multicollinearity. When the predictor variables (a.k.a parameters) are correlated, we call that scenario as Multicollinearity. The presence or absence of Multicollinearity does not give and an indication of our model's accuracy. You can have an idea of the Multicollinearity in your model by running a regression analysis in any statistical software like 'Minitab' or 'SPSS'. In the output, you will see a metric called 'VIF'. It is the short form for the Variance Inflation Factor.VIF points out the variables that are correlated. So if the VIF>10, You can conclude that Multicollineariy affects your model in a bad way and it is better to drop those variables.
This is the way that you can decide whether having uncorrelated parameters in the model make it better. If you need more information on this topic, please visit

| cite | improve this answer | |
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.