It's a class project (however, I want to expand it to make it bigger than just a class project for my own sake) and the professor gives us freedom to choose whatever we want as long as it includes some dataset (deal with them using R) and analysis of it. Since I'm starting to become interested in stock market, algorithmic trading, etc. I was wondering whether I can find some such projects here. My professor now doesn't have some specific project though.


closed as too broad by Nick Cox, John, Nick Stauner, Scortchi Feb 23 '17 at 16:08

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ @DJohnson I deleted the self-study tag as this should be done by the OP if this is the case. "pythh" if this is the case consider adding the self-study tag to your question. $\endgroup$ – Stefan Feb 22 '17 at 18:30
  • 1
    $\begingroup$ To my knowledge there are no such "projects" available here. I am sure you'd find projects on the internet somewhere. Then if you have specific questions regarding data analysis, you could ask them here. Also have a look at the help center. $\endgroup$ – Stefan Feb 22 '17 at 18:35
  • 1
    $\begingroup$ Looking at @DJohnson answer now, he certainly provided some good ideas for a project AND how impress your professor :) $\endgroup$ – Stefan Feb 22 '17 at 18:54
  • $\begingroup$ The question is a bit off topic, but seems harmless enough and some of us have taken the time to answer it, so I am voting to keep it open. $\endgroup$ – Carl Feb 22 '17 at 20:37

Ok...I'll bite. First, there are a multitude of resources out there containing free market data at various levels of detail and composition. For instance:

  • Yahoo! Finance provides easily downloadable past 12 months price information for all US traded stocks and, if you know the underlying CUSIP, you can pull down information about bonds too

  • Robert Shiller's website has a wealth of historic information going back as far as the mid-19th c. http://www.econ.yale.edu/~shiller/data.htm

  • Kenneth French's website has portfolio level information http://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html

  • CRSP - Center for Research in Security Prices has another boatload of data http://www.crsp.com/ although I don't know if it's free

  • iVelocity.com has information about stock options

  • Quandl.com has literally millions of financial time series indexes from an enormous range of sources

It might be cool to mash up a bunch of stock, macroeconomic indexes, etc., to see what kind of relationships emerge.

Next, if you want to impress your professor, don't use ANOVA for the analysis. It has linear, Gaussian assumptions wrt the residuals and it's well known that the tails of financial data are way more extreme than Gaussian. So, try something neat like Hyndman's boosting additive quantile regression as described in this paper, "Forecasting Uncertainty in Electricity Smart Meter." (http://robjhyndman.com/papers/smart-meter-quantiles.pdf). Forget the "electricity" part, it's just a novel methodology that can be applied to financial information. Once the quantiles have been fit, you can assume any underlying probabilistic, extreme value distribution that you like.

Didier Sornette's book "Extreme Financial Risks" is an excellent overview of extreme value theory as applied to financial markets. In addition, his ETHZ website has many more amazingly creative papers well worth browsing (http://www.er.ethz.ch/financial-crisis-observatory.html).

So, hopefully this helps. At a minimum, you have pointers to some useful resources for data. The extreme value theory suggestion may not be that helpful, depending on your comfort level with its highly technical demands.

/** Additional Data Sources **/

  • The St Louis Fed has a large archive of macroeconomic indicators https://research.stlouisfed.org/econ/mccracken/fred-databases/

  • The US Financial Diaries has hhold level information about consumption patterns http://www.usfinancialdiaries.org/

  • The Billion Prices Initiative is described as an, "Academic initiative that uses prices collected from hundreds of online retailers around the world on a daily basis to conduct economic research. This page shows our most recent research leveraging high-frequency price data, as well as the US daily inflation index." http://bpp.mit.edu/

  • 1
    $\begingroup$ Ordinary least squares regression with monthly stock return data is an entirely standard, perfectly defensible thing to do, especially for an undergrad project. (Use heteroskedastic robust standard errors.) $\endgroup$ – Matthew Gunn Feb 22 '17 at 19:28
  • $\begingroup$ (+1) great material in your answer. CRSP is the absolute gold standard for stock price and returns data, and it is not free. If you are at a subscribing university, you may be able to get access to CRSP through wrds-web.wharton.upenn.edu. You may have to go through your school's business school or some finance professor etc... I don't know how it works for undergrads. $\endgroup$ – Matthew Gunn Feb 22 '17 at 19:31
  • $\begingroup$ @MatthewGunn Thanks. Just checked. My university is a subscribing university. $\endgroup$ – pythh Feb 22 '17 at 21:23

There is a problem with your plan. You cannot use ANOVA on stock market data. The statistical distribution involved in stock market data has been in dispute since 1963. That ended recently with someone doing a first principles derivation of the returns that would exist under a variety of circumstances. With the narrow exception of firms being merged out of existence for cash, you cannot use ANOVA.

The distribution of returns, for equity securities that are going concerns, must be some form of transformation of the Cauchy distribution which is then truncated at -100%. You can show that the CDF of the Cauchy distribution is the limit of one of the integrals when the problem is converted to polar coordinates. In any circumstance, the solution is either a Cauchy distribution, or a deformation of the Cauchy distribution.

The difficulty is that the Cauchy distribution lacks a mean, and therefore also lacks a variance. At first this is a strange idea, because you can add numbers up and divide the sample size and therefore you have a sample average. You, of course, have sample averages, but they mean nothing because there is no population average. Because returns are truncated your center of location is the mode and there is no standardized term in use for the scale parameter because truncation disrupts simple descriptors like the inter-quartile range.

It better to think of the variance as a feature of a distribution than as something that must exist, like noses are a feature of most vertebrates. If you see something without a nose, its probably not a vertebrate. This also renders most algorithmic trading methods invalid.

The field will have years of clean-up to do. While you should master ANOVA as it is a very useful tool, do not use it with stocks or anything else that grows at an exponential rate, such as cancer. You do want to master this tool because of its intrinsic value, nonetheless.

  • $\begingroup$ Thank you! I will discuss your answer with my professor. $\endgroup$ – pythh Feb 22 '17 at 21:26
  • $\begingroup$ @pythh if you need direction to further resources, let me know. $\endgroup$ – Dave Harris Feb 22 '17 at 21:27
  • $\begingroup$ Sure! Thank you! (I tried to @ you to reply, but it didn't work) $\endgroup$ – pythh Feb 22 '17 at 21:31

Not the answer you're looking for? Browse other questions tagged or ask your own question.