3
$\begingroup$

We know that if $X$ is positive, then $X^2$ is highly positively correlated with $X$. I've plotted an array of integer numbers from 100 and 110 with the following code:

X = np.arange(100,111)
X_2 = X**2
plt.scatter(X, X_2)
plt.show()

The correlation as computed with the Numpy corrcoef function is pretty high: 0.9999115763553446

However, it is just sufficient to subtract the mean of $X$ from $X$ to decorrelate it with $X^2$:

X_mean = X.mean()
X = X - X_mean
X_2 = X**2
plt.scatter(X, X_2)
plt.show()

The correlation now is 0.0.

So I did the same with $X^3$:

X = np.arange(100,111)
X_mean = X.mean()
X = X - X_mean
X_3 = X**3
plt.scatter(X, X_3)
plt.show()

However, in this case, the centering does not help since the odd powers of the centered $X$ remain negative in the first half of the range and there is thus a positive correlation of 0.9996468005152317 between the centered $X$ and $X^3$. So, how do we decorrelate them?

$\endgroup$
20
  • 6
    $\begingroup$ Regress $X^3$ on $X$ (via least-squares) and use residuals instead of $X^3$. $\endgroup$
    – Michael M
    Aug 26, 2023 at 20:26
  • 1
    $\begingroup$ @MichaelM Care to expand that into an answer? $\endgroup$
    – Dave
    Aug 26, 2023 at 20:53
  • 4
    $\begingroup$ Why not start off with an orthogonal polynomial basis to begin with? If your X variables can take any value between -infinity and infinity, a natural choice is the Hermite polynomials, which in Python are implemented in scipy: docs.scipy.org/doc/scipy/reference/generated/… $\endgroup$ Aug 26, 2023 at 21:15
  • 4
    $\begingroup$ oh and a second question: why do you want to decorrelate them in the first place? $\endgroup$ Aug 26, 2023 at 21:48
  • 2
    $\begingroup$ I echo @JohnMadden 's second comment. What are you going to do once you have decorrelated these? Are they going to be independent variables in a regression? Or what? $\endgroup$
    – Peter Flom
    Aug 26, 2023 at 23:13

3 Answers 3

1
$\begingroup$

If $X$ has multiple values, then for any increasing function $f(X)$, the correlation between $X$ and $f(X)$ will always be positive.

So, there is no transform that you can apply to $X$ (like subtracting the mean) such that for the transformed variable $X'$ the correlation between $X'$ and $f(X')$ becomes zero, except a transformation with a multiplication by zero.

(The multiplication by zero will make that there are no multiple values anymore)


If the correlation is zero then the covariance is zero.

This covariance can be written as

$$\begin{array}{} \text{Cov}[X,f(X)] &=& \frac{1}{n}\sum_{i=1}^n (x_i-\mu_{x}) (f(x_i)-\mu_{f(x)}) \end{array}$$

And shifting up or down the second term with a constant $a$ will not change the sum

$$\frac{1}{n}\sum_{i=1}^n (x_i-\mu_{x}) (f(x_i)-\mu_{f(x)}) = \frac{1}{n}\sum_{i=1}^n (x_i-\mu_{x}) (f(x_i)-\mu_{f(x)}-a)$$

Now, if $f(X)$ is an increasing function of $X$, then we can choose the constant $$a = f(\mu_x)-\mu_{f(x)}$$ such that $$\text{sign}(x_i-\mu_{x}) = \text{sign}(f(x_i)-\mu_{f(x)}-a)$$ and the terms in the sum will be a sum of only non-negative terms

$$\text{sign}\left((x_i-\mu_{x})\cdot(f(x_i)-\mu_{f(x)}-a)\right) = \begin{cases} 0 &\quad \text{if $x_i = \mu_{x}$} \\ 1& \quad\text{else}\end{cases}$$

Therefore, the summation used to compute the covariance, can be written as a sum of terms that are all non-negative, and the final sum will be non-negative, where equality to zero only occurs when for all values we have $x_i = \mu_x$, ie when all values are the same.

Example

In the comments you asked

Can you explain why $\text{sign}(x_i-\mu_{x}) = \text{sign}(f(x_i)-f(\mu_{x})) $

possibly the following illustration may help

example

To compute the covariance you multiply for each point the values on the horizontal axis with the values on the vertical axis (and take the average).

The difference between the left and right image is only that we moved the points up such that the function (defining the points) crosses the origin. This shift leaves the result from the computation unchanged (the constant multiplied with the $x_i-\mu_x$ will be on average zero), but now all the terms will be a product of two negative numbers or two positive numbers and we can clearly see that the end result needs to be positive.

This shift can be made for any increasing function, no matter what the underlying X values are, with the only exception when the points are all the same value.

$\endgroup$
5
  • $\begingroup$ Why did you subtract the constant $a$ in the second term of the covariance and not the first? The idea is to have $X^{\prime}=X−a$. Also, in your answer, you argue that "[...] there is no transformation [...]," but it seems you are only demonstrating that there is no constant that can be subtracted such that for the transformed variable $X^{\prime}$ the correlation between $X^{\prime}$ and $f(X^{\prime})$ becomes zero. $\endgroup$
    – ricber
    Aug 28, 2023 at 15:33
  • $\begingroup$ @ricber The use of the constant 'a' is to transform the function $f$ (by adding a constant) and not to transform $X$. The reason that I introduced that constant is to get a quick and easy proof that for any increasing function $f(\cdot)$, and your case $f(x)=x^3$ is a just a special case, we have that the correlation between $x$ and $f(x)$ is always positive, no matter what you do to the set of points $x_i$ (except making all points the same value, like multiplying with zero). $\endgroup$ Aug 28, 2023 at 16:23
  • $\begingroup$ Can you explain why $\text{sign}(x_i-\mu_{x}) = \text{sign}(f(x_i)-f(\mu_{x}))$? $\endgroup$
    – ricber
    Aug 29, 2023 at 8:11
  • $\begingroup$ @ricber if f(x) is an increasing function of $x$ then it is a line/curve that crosses x-axis only once. By shifting the function up/down then we get it to make that point where it crosses the y-axis equal to the point where it crosses the x-axis. $\endgroup$ Aug 29, 2023 at 8:53
  • $\begingroup$ Thank you for the illustration! Now it's much clearer! $\endgroup$
    – ricber
    Aug 29, 2023 at 11:56
1
$\begingroup$

You have $x=(-5,-4,-3,-2,-1,0,1,2,3,4,5)$ and $y = x^3$ where cubing is just term by term: $((-5)^3, (-4)^3, (-3)^3, \ldots).$

Fitting the line $y = a+bx$ by least squares you get $a=0$ and $b=89/5 = 17.8.$

Thus $x^3-17.8x$ is uncorrelated with $x.$

If you go from $-6$ to $+6$ rather than from $-5$ to $+5,$ you'll get some other number than $17.8.$

If the least-squares line is $y=a+bx$ then $y-(a+bx)$ is uncorrelated with $x.$

$\endgroup$
1
$\begingroup$

This kind of mathematical collinearity is not worth a lot of effort to solve as it has no consequences other than when you try to interpret a coefficient in isolation. In general, collinearity is harmful mainly when predicting on new data having collinearities that are inconsistent with the training data collinearities.

Many regression programs use the QR decomposition internally, so any numerical problems are fixed without the user needing to pay attention to what's under the hood. QR is reversed to get the final coefficients and standard errors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.