# Correlation between linear regression coefficients

Can we get correlation between linear regression coefficients?

I have real data (time series) with different realizations representing the same phenomenon: $\{x_j(t)\} = x_1(t), \dots, x_N(t)$. I model each one with a linear regression model: $f_j(t)=\displaystyle \sum_{i=0}^dc_{ij}\phi_i(t)$, $(j = 1, \dots, N$). $\phi_i(t)$ are Legendre polynomials and are orthogonal.

After estimating the $c_{ij}$ I found correlations between these coefficients wrt $j$, i.e. if I plot (see fig. below) the different $c_{ij}$ function of $j$ I find that $c_{mj}$ and $c_{lj}$, for example, are linear combinations of each other. This happens for all coefficients. Is this normal? What does it mean?

Correlation between linear regression coefficients can occur when terms in your model have very similar effects on the targeted output (in your case, this targeted output is $f$). For example, perhaps an increase in 20% in terms 1 and 3 both yield a 5% increase in $f$.

In such a case, the fitting routine cannot determine which term is having the biggest effect on $f$, and so can end up confusing several terms. This leads to the coefficients of these terms being highly correlated.

The correlation between the coefficients of different Legendre polynomials may indicate that your behavior is better described by a simpler model.

Let us assume, for example, that your $t$ are very small, such that higher-order terms are negligible. Then we might expect that your Legendre polynomials will reduced to alternating 'linear' and 'constant' terms.

From Wikipedia, the first four Legendre polynomials are:

$1$

$x$

$\frac{1}{2}(3x^2-1)$

$\frac{1}{2}(5x^3 - 3x)$

For small $x$, the first and third (and fifth, seventh, etc.) terms will reduce to a constant. The other terms will reduce to a constant times $x$. The fitter will not know how to differentiate between the multiple constants or multiple linear terms, and you will end up with very high correlations between your coefficients.

• why will first, third, fifth, etc. terms be reduced to a constant and others (even terms) to a constant times x? Oct 23, 2016 at 16:00
• @MedNait : Let's say that x = 1E-3 . (Perhaps the OP is dealing with short times measured in fractions of a second.) Then x^2 = 1E-6, and x^3 = 1E-9, etc. This means that a term like 3x^3 - 1 would, for x = 1E-3, reduce to 3E-9 - 1. This is essentially the same thing as -1! For terms like 5x^3 - 3x, the argument is very much the same -- the term 5x^3 is so small compared to 3x for x= 1E-3, that we can simply ignore it. Oct 23, 2016 at 16:04
• I added a figure for a better illustration. Oct 23, 2016 at 16:05
• @MedNait : Ah. So there are two kinds of correlation that I could see, here. 1) That the coefficients were correlated according to the model diagnostics. 2) That for many different models f, the coefficients increase and decrease in unison across models. If you're talking about (2), then this could simply be a matter of the effect that you are measuring getting larger and smaller across samples. All coefficients for a given model may then be scaled by some constant to accommodate this fluctuating size. Oct 23, 2016 at 16:12
• Actually what differentiates $f_j(t)$ is the coefficients $c_{ij}$. I don't know if we can say we have different models. Basically, I have different realizations $x_j(t)$ of the same phenomenon and I model each one with corresponding $f_j(t)$ (that defers from others just by the estimated coeff. $c_{ij}$. Oct 23, 2016 at 16:18