# Correlation between two variables of unequal size

In a problem I am working on, I have two random variables, X and Y. I need to figure out how closely correlated the two of them are, but they are of different dimensions. The rank of the row space of X is 4350, and the rank of the row space of Y is substantially larger, in the tens of thousands. Both X and Y have the same number of columns.

I need a measure of correlation between the two variables, and Pearson's r requires X and Y to have equal dimension (at least R requires the two r.v.'s to be).

Do I have any hope of doing a correlation between these two, or should I find some way of pruning off observations from Y?

 EDIT


Adding information from the comments, which should be in the question.

I suppose I forgot to mention this. X and Y are stock prices. Company X has been public for a much shorter time period than Y. I wanted to tell how correlated the prices of X and Y are. I could definitely get a correlation for the period of time that X and Y both exist. I wanted to know if knowing the stock prices for several extra years of Y that X did not exist yielded me any additional information.

• This does not sound like you have observations (or "cases") on which you observe both an X and a Y realization. How do you find out which X is associated to which Y? – Stephan Kolassa Oct 14 '10 at 7:01
• I suppose I forgot to mention this. X and Y are stock prices. Company X has been public for a much shorter time period than Y. I wanted to tell how correlated the prices of X and Y are. I could definitely get a correlation for the period of time that X and Y both exist. I wanted to know if knowing the stock prices for several extra years of Y that X did not exist yielded me any additional information. – Christopher Aden Oct 14 '10 at 7:10
• @Christopher I'd recommend that you update your question to reflect your above comment. Also, for correlation to be meaningful, more than just equal dimensions are required; the actual measurements have to come from the same cases, which in your case is presumably the same time points. – Jeromy Anglim Oct 14 '10 at 8:13
• I second Jeromy's comment on updating the question... – Stephan Kolassa Oct 14 '10 at 8:53
• Another question: you mention that X and Y have the same number of columns. Would that be one each? Or do you have multiple series for both X and Y (prices at different stock exchanges or some such)? – Stephan Kolassa Oct 14 '10 at 9:00

## 7 Answers

No amount of imputation, time series analysis, GARCH models, interpolation, extrapolation, or other fancy algorithms will do anything to create information where it does not exist (although they can create that illusion ;-). The history of Y's price before X went public is useless for assessing their subsequent correlation.

Sometimes (often preparatory to an IPO) analysts use internal accounting information (or records of private stock transactions) to retrospectively reconstruct hypothetical prices for X's stock before it went public. Conceivably such information could be used to enhance estimates of correlation, but given the extremely tentative nature of such backcasts, I doubt the effort would be of any help except initially when there are only a few days or weeks of prices for X available.

• Clarification: I didn't mention GARCH to deal with the missing data problem (which of course would not make sense) - but to improve on a simple calculation of correlation between the time series at times where both exist. – Stephan Kolassa Oct 14 '10 at 14:12
• @Stephan: OK. I mentioned it mainly to show I wasn't ignoring you! – whuber Oct 14 '10 at 14:24
• Thank you, whuber. This is in line with what I was looking for. I don't think the backcasting will be of much use (or feasibility) to add a couple extra weeks of X when the mutual time frame between X and Y is about 16 years already. – Christopher Aden Oct 14 '10 at 15:36
• @Christopher: !! With 16 years (of daily closings?) you have enough data not only to find a correlation, but also to explore how it has been changing over time. (This I believe is the spirit of @Stephan Kolassa's reply.) – whuber Oct 14 '10 at 16:08
• I agree. Using techniques to figure out what values X would've taken prior to its IPO seems prone to error. I might also question the relevance of data that's 16 years old to predict modern trends. – Christopher Aden Oct 14 '10 at 22:20

So the problem is one of missing data (not all Y have a corresponding X, where correspondence is operationalized via time points). I don't think there is much to do here than just to throw away the Y you don't have an X for and calculate the correlation on the full pairs.

You may want to read up on financial time series, though I don't have a good reference handy at this point (ideas, anyone?). Stock prices often exhibit time-varying volatilities, which can be modeled, e.g., by GARCH. It is conceivable that your two time series X and Y exhibit positive correlations during periods of low volatility (when the economy grows, all stock prices tend to increase), but negative correlations when overall volatility is high (on 9/11, airlines tanked while money fled to safer investments). So just calculating an overall correlation may be too dependent on your observation time frame.

UPDATE: I think you may want to look at VAR (vector autoregressive) models.

@Jeromy Anglim specified this correctly. Having the extra information when only one of the time series existed would provide no value here. And in principle, the data should be sampled at the same time for it to be meaningful using conventional correlation measures.

As a more general problem, I would add that there are techniques to deal with irregularly spaced time series data. You can search for "irregularly spaced time series correlation". Some of the recent work has been done on "Realized Volatility and Correlation" (Andersen, Bollerslev, Diebold, and Labys 1999) using high-frequency data.

Given the extra information in your comments I'd recommend looking at two correlations. The first would be the common time periods that the companies were both around. So, if one was around 2 years earlier you'd just drop that data and look at the rest. The second would be the relative time periods. In the second one you're not correlating actual time but time measured since the company went public.

The former would be strongly influenced by general economic forces shared within the same time period. The latter would be influenced by properties shared by companies as they change after the IPO.

Another way to solve such a problem is to impute the missing data for the shorter series using a time series model which may or may not make sense in a particular context.

In your context, imputing the stock prices into the past would mean that you are asking the following counter-factual question: What would be the stock price for company X had it gone public n years in the past instead of when it actually went public? Such a data imputation could potentially be done by taking into account stock prices of related companies, general market trends etc. But, such an analysis may not make sense or may not be needed given the goals of your project.

Well a lot depends on the assumptions you make. If you assume that the data is stationary then more data for series one will give you abetter estimate of its volatility. This estimate can be used to improve the correlation estimate. So the follwoing statment is incorrect:

"The history of Y's price before X went public is useless for assessing their subsequent correlation"

• I thgought about this. In theory may work, but will be very unrobust, so better to avoid. – kjetil b halvorsen Mar 3 '17 at 8:38

This sounds like a problem for a machine learning algorithm. Therefore, I would try to figure out a set of features which describe a certain aspect of the trend and train on that. The whole machine learning theory is a bit to complex for this answer-box, but it would be useful for you to read into it.

But honestly, I think that already exists out there. Where money can be made, people put their mind in it.