4
$\begingroup$

I have collected data from Twitter, Google and Wikipedia to look for any correlations (and perhaps predictive value) against the stock market.

I have collected daily Twitter, Google and Wikipedia data for 148 days. In regression analysis, I am using a dummy variable, "sector". All 100 companies from which I have collected data can be classified in one of 6 industry sectors.

Now, I started thinking that a regression analysis may be much stronger for companies that have had higher news coverage. It is acceptable to think that if there's much news about a company, people tend to tweet more about that company.

I have collected news headlines mentioning each of the 100 companies over the same 148 days. There are many days for many of the companies on which there is no news at all, so trying to correlate non-news days to number of tweets for instance, doesn't seem logical.

I have now aggregated all the headlines, for instance: during the 148 days, there were 846 headlines about Apple, 137 about Microsoft and 0 about Abbott Laboratories. I'm now thinking to of adding a dummy variable to the dataset, something like dNews10, dNews100, meaning companies with less than 10 news articles, 10 to 100 articles, etc.

Would this be an acceptable variable to add to the analysis? Usually, as I have been taught, dummies are properties to the research subject (man: hight, sex, eye-color, etc.). I don't know if news coverage would qualify.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Given that you have time series data, I think you want to do time series analysis. I added that tag so experts on the topic may see it. Regular regression with time series data is fraught with pitfalls. $\endgroup$
    – Peter Flom
    Commented Aug 5, 2012 at 12:19

1 Answer 1

Reset to default
1
$\begingroup$

Company tweets do vary with time and so have time dependence as well as dependence on covariates. But it is possible to model covariates in time along with an ARMA structure in such a model. Instead of using them continuously you choose to create categories. There is nothing wrong with that (except that some quantitative information is lost). I would keep track of these covariates as a function of time as recent news reports will have more influence and relevance than old ones. Treat the problem as a time series forecasting problem rather than as a simple regression problem.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.