4
$\begingroup$

Let's say there are two random variables (for example, two time series data of S&P500 and a stock) and their correlation is 0.95. What is the best way to reduce the correlation between these two variables to 0.7, while keeping one as it is? One approach that comes to mind is to add noise (error terms). But how would I know what should the mean standard deviation for the noise? I would appreciate if one could show how to do this in R.

$\endgroup$
  • $\begingroup$ Not quite sure what you're asking here - if two variables are correlated, then they're correlated. You can change them to whatever you want (aka adding noise), but then they're different variables. $\endgroup$ – norvia May 14 at 20:45
  • $\begingroup$ I want to add noise only to one of those random variables and keep the other one as it is. $\endgroup$ – AK88 May 14 at 20:47
  • 2
    $\begingroup$ Yes that helps. And it shows you are not viewing these as time series, but only as paired data. $\endgroup$ – whuber May 14 at 20:51
  • 1
    $\begingroup$ @AK88 in the third method overall variance remains the same. There’s no additional noise in any of methods I suggested $\endgroup$ – Aksakal May 15 at 19:46
  • 1
    $\begingroup$ @AK88, another thing to be careful with if you're doing backtesting: watch the higher moments, such as kurtosis of a new variable. In one of my examples the tail thinned a lot, this has huge implication on the metrics such as VaR, you'll be underestimating the tail risk, and also increased the "benefit" of diversification by lower correlation. $\endgroup$ – Aksakal May 15 at 20:15
1
$\begingroup$

So if $\rho_N$ is the desired Pearson correlation, $\rho_0$ is the current correlation with $\rho_N\le \rho_0$, and $\epsilon$ is our uncorrelated noise:

$\frac{cov(X+\epsilon,Y)}{\sqrt{var(X+\epsilon)\sigma^2_Y}}=\rho_N$

Solving for variance of $\epsilon$

$\sigma^2_\epsilon=\frac{cov(X,Y)^2}{\rho_n^2\sigma_Y^2}-\sigma^2_X=\sigma_X^2\bigg(\frac{\rho_0^2}{\rho_N^2}-1\bigg)$

So, for your example, you'd be setting the variance of the noise to be $\approx .357\sigma_X^2$.

(may want to double check my algebra here)

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks! Just to make sure that I got this right -- are you saying that $cov(X+\epsilon,Y) = cov(X,Y)$ $\endgroup$ – AK88 May 14 at 21:53
  • 1
    $\begingroup$ Yep - $cov(X+\epsilon, Y)=cov(X, Y)+cov(\epsilon, Y)=cov(X,Y)$ because $\epsilon$ is uncorrelated with $Y$. $\endgroup$ – norvia May 14 at 21:57
  • 1
    $\begingroup$ We're adding the noise to $X$, not $Y$ here, so it should be $\sigma_X^2$. Intuitively the equation is saying you need scale up the size of the noise with the variance of $X$. If $X$ has a huge variance, then you're going to need appropriately larger noise to impact it. $\endgroup$ – norvia May 14 at 22:10
  • 1
    $\begingroup$ You make some hidden but crucial assumptions: you need $\epsilon$ to be uncorrelated with $X$ as well as uncorrelated with $Y.$ As far as how to do this in R goes, see the solution at stats.stackexchange.com/a/152034/919. $\endgroup$ – whuber May 14 at 22:11
  • 1
    $\begingroup$ sorry, you are right. got messed up in my derivation. Thanks! $\endgroup$ – AK88 May 14 at 22:12
4
$\begingroup$

Given the intended application, you might be interested in creating realistic modifications of the series of data.

Overview

This is easier to do than you might think. (See the three-line function decorrelate in the code below.) The idea is to generate the "noise" series realistically, according to any model you like. (You can even use actual data, such as a series for another stock or a portion of the series for the same stock at a different time.) The desired amount of correlation reduction determines what multiple of the noise series you need to add in order to achieve the intended reduction in correlation.

Analysis

To be explicit, let's suppose you have paired series $(x_t,y_t)$ and can produce a third series $\epsilon_t$ to use for the noise. Let the correlation coefficient of $(x_t,y_t)$ be $\rho$ and suppose you wish to obtain a new series $(y^*_t)$ (with the same average value as $(y_t)$) whose correlation with $(x_t)$ is $\kappa\rho.$ I assume $|\kappa|\lt 1:$ that is, you intend to reduce the correlation.

As a technical preliminary, intended to ensure you will be able to obtain an answer, we need first to "decorrelate" the noise from the original series. This is readily done with ordinary least squares regression: fit the model $E[\epsilon] = \beta_0 + \beta_x x + \beta_y y$ to the data and replace the series $(\epsilon_t)$ with the residuals of that model, found by subtracting its fitted values from the original values. Let $(e_t)$ be the residual series. (If you're truly unlucky, this series could be entirely zero: that would mean the "noise" was originally a linear combination of $(x_t)$ and $(y_t).$ In such a case you will have to come up with a different noise series and try again.) As a bonus, including the constant term $\beta_0$ in the model guarantees the average value of $(e_t)$ is zero, so that adding the noise does not change the overall level of the data.

The new series will be obtained by adding some positive multiple $\lambda$ of the noise to either $(y_t)$ (when $\kappa \ge 0$) or to $(-y_t)$ (when $\kappa \lt 0,$ which means you want to reverse the direction of correlation). That is, the solution will be

$$(y_t^*) = (\operatorname{sgn}(\kappa) y_t + \lambda e_t).$$

The value of $\lambda$ is found by comparing the correlation coefficients. The requirement (when $\kappa \gt 0$) is

$$\operatorname{Cor}(x_t, y_t^*) = \frac{\operatorname{Cov}(x_t, y_t + \lambda e_t)}{\sqrt{\operatorname{Var}(x_t) \operatorname{Var}(y_t+\lambda e_t)}} = \kappa \rho .$$

Writing $\sigma^2$ for the variance of $(x_t),$ $\tau^2$ for the variance of $(y_t),$ and $\psi^2$ for the variance of $(e_t),$ this equation becomes

$$\frac{\rho\sigma\tau}{\sigma \sqrt{\tau^2+\lambda^2\psi^2}} = \kappa \rho.$$

If $\rho=0$ there's nothing to do, and when $\rho\ne 0$ this conveniently simplifies (when squared) to

$$\frac{1}{1+\lambda^2\psi^2/\tau^2} = \kappa^2,$$

with the unique solution

$$\lambda = \frac{\tau}{\psi}\sqrt{\frac{1}{\kappa^2}-1}.$$

It's easy to verify this works for negative $\kappa$ as well.

To implement this in R, I offer the decorrelate function at the beginning of the code below. At line 2, it performs the decorrelation step using lm and residuals. It invisibly divides by $\psi$ using the scale function (also on line 2), but otherwise directly reflects the foregoing formula (line 3). For convenience, if you have no definite noise series $(\epsilon_t)$ to supply, it will create one randomly (out of Normal white noise, line 1).


Examples

First, here is the solution using the default white noise.

Figure 1 showing scatterplot matrix of time, x, y, and noisy y

In this scatterplot matrix, the original series $(x_t)$ is at the top upper left plotted against time. It exhibits noticeable positive serial correlation: it's not jagged and random, but smoothly undulating. Beneath it are $(y_t)$ (also serially correlated) and then--this is the first solution, down in the lower left corner--the noisy version of $(y_t).$ (The rest of the matrix displays correlations among the series as scatterplots. The colors consistently show the times.)

The original correlation of $\rho=0.95$ has been reduced to precisely $\kappa \rho = 0.60.$ However, there's a problem: qualitatively, the noisy series doesn't look much like $(y_t).$ It's too jagged.

We solve this by using a noise series that has (approximately) the same serial correlation as $(y_t).$ Here's what the second solution looks like:

Figure 2 showing modified solution in the same way as Figure 1

Look at the lower left corner. Once again, the correlation of this modified series $(y_t^*)$ with $(x_t)$ is precisely $\kappa \rho=0.60.$ If that seems like an improvement, then this method is probably what you want to use.


Code

Here is the R code that generated the data and the figures. You can modify its parameters to experiment. When you have real data, you only need the code starting at "Reduce their correlation" (as well as the decorrelate function, of course).

The mysterious part (in the middle) is the generation of a series $(x_t,y_t)$ with specified correlation. This is carried out by applying the algorithm to the series $(x_t,x_t),$ for which the correlation is $1,$ and reducing that correlation by an amount $\rho.$ I am pleased with this trick :-).

# Multiply the absolute correlation between `x` and `y` by a factor `kappa`
# by adding noise to `y`.  Returns the noisy `y`.
#
decorrelate <- function(x, y, kappa, noise) {
  if (missing(noise)) noise <- rnorm(length(x))
  eps <- c(scale(residuals(lm(noise ~ x + y))))
  y * sign(kappa) + eps * sd(y) * sqrt(1/kappa^2 - 1)
}
#
# Create MA time series data.  A Gaussian window of length 2w+1 is run over an iid 
# standard normal series to create a sequence of `n` values.
#
ts.create <- function(n, w) convolve(rnorm(n+2*w), dnorm((-w):w, 0, 2), type="filter")
#
# Create correlated data.
# set.seed(17)
rho <- 0.95
n <- 150
w <- 19
x <- convolve(rnorm(n+2*w), dnorm((-w):w, 0, 2), type="filter")
y <- decorrelate(x, x, rho, noise=ts.create(n, w))
#
# Reduce their correlation.
#
rho.new <- 0.60
y.star <- decorrelate(x, y, rho.new / rho, noise=ts.create(n, w))
#
# Graph the series and show their scatterplots.
#
f <- function(x, y) {
  if (diff(range(x))+1==length(x)) {cex=0.7; lines(x,y,col="Gray")} else cex=1
  points(x,y, pch=19, cex=cex, col=hsv(seq(0, 5/6, length.out=length(x)), .9, .9, .5))
}
pairs(cbind(time=seq_along(x), x, y, `noisy y`=y.star), upper.panel=NULL, lower.panel=f)
| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

First, since OP's problem is with relation to stock and index returns correlation, I must note that one thing to be aware of is that if the stock, such as AAPL, is a part of index such as SPX, then the problem is a little more complex, because any change to AAPL will spill into SPX index.

So, let's assume that the stock in question is not a constituent of SPX index. In this case I offer you two and half methods. First two would preserve the index as is, and the last one will not, but it will not add external noise either.

mix-in noises with one variable staying the same

I can transform one of the variables in such a way that the other stays the same but correlation changes. It's a simple mixing in the noise from first variable into the second. This is how you do it.

Get new series Y as from original series X by solving for a and b: $$y_1=x_1\\ y_2=ax_1+bx_2\\var[y_2]=var[x_2]=\sigma_2^2\\cov[x_1,x_2]=\rho\sigma_1\sigma_2\\cov[y_1,y_2]=c\sigma_1\sigma_2$$

Write down variance and covariance of $y_2$: $$var[y_2]=a^2\sigma_1^2+b^2\sigma_2^2+2ab\rho\sigma_1\sigma_2\equiv\sigma_2^2$$ $$cov[y_1,y_2]=a\sigma_1^2+b\rho\sigma_1\sigma_2=c\sigma_1\sigma_2$$ $$a\sigma_1+b\rho\sigma_2=c\sigma_2$$

You have two nonlinear equations and two unknowns a,b. You can solve this analytically or like me, a lazy guy, numerically with Excel solver, which got me:

  • a = 1.375
  • b = -2.0213

and the following plot of new series, where you can see that SPX returns remained the same and only AAPL returns changed while mean and variance were preserved:

enter image description here

Here's the first four moments after the transformation: enter image description here

Note, that only kurtosis has changed significantly, yet it's a very important moment in portfolio management, for it is responsible for a lot of tail risk. In this case, the tail of the second variable thinnedconsiderably.

re-ordering variables

Second method is to not change the observations at all, but instead rearrange the observations in variables in such a way that correlation changes. Consider this: if I take random samples from both series, and calculate the correlation between these samples, then correlation will be zero. So, simple rearranging the order in variables will change correlation.

Advantage of this method is that variances remain the same, and overall distributions remain intact in the dataset. Since, you work with stock return, there's no autocorrelation, so rearranging doesn't impact this aspect at all, at least in theory. I don't know whether this method can generate drastic changes in correlation such flipping its sign like in the example above.

shuffling noise around

Instead of adding noise, let's push it around! We'll shuffle noise between two variables so that the correlation is what we want while preserving variances of each variable. Here's the final result for S&P 500 and Apple returns in 2020 Jan-Apr, where original correlation was 94% which I changed into -70%:

enter image description here

I used PCA and Cholesky decomposition to do this:

  • apply PCA to destandardized original series X to get the uncorrelated series F
  • standardize PC factors F to get unit variance series U
  • apply Cholesky decomposition to standardized uncorrelated series U to obtain correlated series Y with unit variance
  • destandardize Y to get to the series with original means and variances X'

This looks a bit intimidating, but you don't have to do this manually, and can use software. We don't introduce external noise. We only push the noise that exists in the data already from one place to another. Total variance of the data set remains the same as well as some other characteristics such as means.

Here's the first 4 moments after the transformation: enter image description here

You can see how the higher moments change, especially, kurtosis. If the stock returns were normal, then this wouldn't have happened. However, the stock return usually have heavier tails than normal. So, watch out for thinner tails if you are analyzing tail risk metrics.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I like the last approach of shuffling subsets of the data, as it can preserve important aspects of the original data. Adding noise could get you into situations with impossible values, like if you add noise and wind up with negative time or weight, for example. Shuffling one entire vector will, on average, get you zero correlation, while shuffling some random subset of the vector will lower the correlation by a smaller amount. $\endgroup$ – Nuclear Wang May 15 at 16:51
  • $\begingroup$ @NuclearWang, I need to figure out the algorithm for shuffling :) thinking about it $\endgroup$ – Aksakal May 15 at 16:51
  • $\begingroup$ I'm not sure of a surefire way to hit a target correlation value, but it should be easy enough to do empirically. For a vector of N values, randomly select some proportion p of indices to shuffle. Now just put those pN values in random order, and put the reordered values back into the indices you selected in the first place. For p =0, you'll have no shuffling and the original correlation value, for p =1, you'll have complete random shuffling and zero correlation (on average). You should be able to find any target correlation in between by varying p. $\endgroup$ – Nuclear Wang May 15 at 17:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.