Correlation between two variables of unequal size

Question

In a problem I am working on, I have two random variables, X and Y. I need to figure out how closely correlated the two of them are, but they are of different dimensions. The rank of the row space of X is 4350, and the rank of the row space of Y is substantially larger, in the tens of thousands. Both X and Y have the same number of columns.

I need a measure of correlation between the two variables, and Pearson's r requires X and Y to have equal dimension (at least R requires the two r.v.'s to be).

Do I have any hope of doing a correlation between these two, or should I find some way of pruning off observations from Y?

Answer 1

No amount of imputation, time series analysis, GARCH models, interpolation, extrapolation, or other fancy algorithms will do anything to create information where it does not exist (although they can create that illusion ;-). The history of Y's price before X went public is useless for assessing their subsequent correlation.

Sometimes (often preparatory to an IPO) analysts use internal accounting information (or records of private stock transactions) to retrospectively reconstruct hypothetical prices for X's stock before it went public. Conceivably such information could be used to enhance estimates of correlation, but given the extremely tentative nature of such backcasts, I doubt the effort would be of any help except initially when there are only a few days or weeks of prices for X available.

Answer 2

So the problem is one of missing data (not all Y have a corresponding X, where correspondence is operationalized via time points). I don't think there is much to do here than just to throw away the Y you don't have an X for and calculate the correlation on the full pairs.

You may want to read up on financial time series, though I don't have a good reference handy at this point (ideas, anyone?). Stock prices often exhibit time-varying volatilities, which can be modeled, e.g., by GARCH. It is conceivable that your two time series X and Y exhibit positive correlations during periods of low volatility (when the economy grows, all stock prices tend to increase), but negative correlations when overall volatility is high (on 9/11, airlines tanked while money fled to safer investments). So just calculating an overall correlation may be too dependent on your observation time frame.

UPDATE: I think you may want to look at VAR (vector autoregressive) models.

Answer 3

@Jeromy Anglim specified this correctly. Having the extra information when only one of the time series existed would provide no value here. And in principle, the data should be sampled at the same time for it to be meaningful using conventional correlation measures.

As a more general problem, I would add that there are techniques to deal with irregularly spaced time series data. You can search for "irregularly spaced time series correlation". Some of the recent work has been done on "Realized Volatility and Correlation" (Andersen, Bollerslev, Diebold, and Labys 1999) using high-frequency data.

Answer 4

Given the extra information in your comments I'd recommend looking at two correlations. The first would be the common time periods that the companies were both around. So, if one was around 2 years earlier you'd just drop that data and look at the rest. The second would be the relative time periods. In the second one you're not correlating actual time but time measured since the company went public.

The former would be strongly influenced by general economic forces shared within the same time period. The latter would be influenced by properties shared by companies as they change after the IPO.

Answer 5

Another way to solve such a problem is to impute the missing data for the shorter series using a time series model which may or may not make sense in a particular context.

In your context, imputing the stock prices into the past would mean that you are asking the following counter-factual question: What would be the stock price for company X had it gone public n years in the past instead of when it actually went public? Such a data imputation could potentially be done by taking into account stock prices of related companies, general market trends etc. But, such an analysis may not make sense or may not be needed given the goals of your project.

Answer 6

Well a lot depends on the assumptions you make. If you assume that the data is stationary then more data for series one will give you abetter estimate of its volatility. This estimate can be used to improve the correlation estimate. So the follwoing statment is incorrect:

"The history of Y's price before X went public is useless for assessing their subsequent correlation"