# Help needed with correlating two datasets

First cross-validated question so please be gentle :o)

I have two datasets all gathered and managed in 'R'...

Dataset 1 - News Corpus. Contains 3,270 entries from the period 1/Apr/13 to 31/Mar/14. There are often multiple stories on any one day, and indeed days with no stories at all (which I believe makes for an incomplete time series and problems). The dataset structure is;

Date - (a date)
Domain - (a string) with 8 levels i.e. there are 8 web domains
DomainType - (a string) with 4 levels e.g. "other news" or "technology news"
Sentiment_Title - (a numeric) a score that currently sits in range -4:4
Sentiment_Description - (a numeric) a score that currently sits in range -6:7
Sentiment_Body - (a numeric) a score that currently sits in range -53:146
CCAT - (logical)
ECAT - (logical)
GCAT - (logical)
MCAT - (logical)


DOWNLOAD corpusData.csv from Mega

Dataset 2 - Bitcoin Market Data. 365 day time series of weighted price, volume and intra-day spread for four different exchanges.

The Problem What I really want to know is which features (if any) of dataset 1 (the corpus) are significantly related to the time series and how. I guess the time series also needs leads and lags applied to know which direction any relationship goes and how far away from the story publication date that relationship lays.

I have spent a couple of weeks applying the very basic stats knowledge I have to the task and have spent a couple of hours with a post-grad stats support group who also proved unable to find a method that could be readily applied.

I (we) looked at basic Pearson's and Spearman's, moved on to look at linear regression and generalised linear models and so far there appears to be issues with the residuals that makes the output bunkum apparently. I believe vector-autoregression could also be applied but we are way off into realms I just don't understand yet.

The Question Given the datasets (and, ideally R) can anyone suggest or indeed offer up an approach to solving my problem? Even better some simple explanation of how to interpret the results of any such approach.

• I do not have expires in this area, so I will only leave a suggestion: would you consider breaking this into several multivariate time series analyses? That is, use multivariate time series analysis (this must exist, also see the microarray gene expression literature for this) on each aspect of the bitcoin data set? This looks like it would only be 12 time series. Also, you could maybe look into canonical correlation analysis to at least explore what potential relationships there are. CCA would ignore the time series structure, but from an exploratory standpoint, could be useful. – Moose Jun 2 '15 at 10:42

## 1 Answer

Vector Autoregression (VAR) (not to be confused with Value at Risk, VaR) allows you to deal with your multiple observations on your bitcoin data, and will model lagged dependencies of volume on day $t$ on volume, spread and price on day $t-1$. So this seems like a start.

In addition, there really is no reason why VAR should not include eXternal variables that are not forecasted by themselves, but should only influence your target variables of interest. Check your favorite software package on how to add such variables to a VAR model, to yield a VARX model. (I am familiar with ARIMAX models, which are ARIMA models including eXternal variables, but not with the VAR analogue, so the term "VARX" may or may not exist.)

In your particular case, you could simply concatenate each day's stories and extract a long string of indicator variables showing whether a term appeared on a given day in any story. That might be a start. I am not familiar with VARX models that respect your inherent hierarchical structure, where terms are grouped within stories, and it may even make a difference whether a term appears only once in a given day, versus ten times in ten different stories.

Finally, if you have many terms, your regressor matrix will be big and sparse. You may need to resort to some kind of regularisation to handle this, but I again don't know how to do this.

I hope this answer at least gave you a couple of terms (like VAR, VARX, hierarchical VAR, regularisation in VARX, ...) that you can fruitfully google. Good luck!

• Thanks for that Stephen. The more I read the more VAR appears to be the right approach (or at least a suitable one). I'm curious by your "concatenate each day's storys" idea - I was thinking I would need to collapse any stories for say a given domain on a day (and perhaps create a single sentiment score for that domain) - I'm not sure this is what you mean though. – BarneyC Jun 4 '15 at 8:47
• Yes, something like this is what I had in mind. This approach would separate occurrences of a single term in multiple domains, which looks good, but you still likely have a huge matrix of covariates. – Stephan Kolassa Jun 4 '15 at 9:07