Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 6 down vote favorite
13

Alternatively, to predict foreign exchange markets. I know this can get pretty complicated, so as an introduction, I'm looking for a simple prediction algorithm that has some accuracy.

(It's for a M.Sc. university project that lasts four months)

I've read that a multi-layer neural network might be useful. Any thoughts on that? In addition, semantic analysis of social media may provide insight into market behavior which influences the stock market. However, semantic analysis is a bit outside the scope of the project at the moment.

share|improve this question
10  
+1 for the chutzpah – user603 Jan 20 '12 at 15:37
7  
Mine, but for obvious reasons I'm keeping it all to myself! – babelproofreader Jan 20 '12 at 15:47
5  
If one believes the efficient-market hypothesis, it is impossible to consistently achieve higher than average market returns (without insider knowledge), which is built into current rates/prices. A lot of people disagree with this, but almost everyone agrees that it is true for a casual investor. In other words, a 3 line model based off of rand() is probably almost as good as the typical investor :P – rm999 Jan 20 '12 at 16:12
3  
It seems unlikely that anyone would be willing to share an algorithm that has any out-of-sample accuracy. Except, perhaps, some published academic work where the anomaly is small and doesn't cover transaction costs. – NPE Jan 20 '12 at 16:47
2  
For academic work, it might be more worthwhile to model the prices rather than try to predict it. Prediction will probably prove to be unsuccessful, but modeling might at least provide some insight into how things actually work, and theoretically may be extended to prediction. – highBandWidth Jan 20 '12 at 17:31
show 8 more comments

6 Answers

up vote 11 down vote accepted

As babelproofreader mentioned, those that have a successful algorithm tend to be very secretive about it. Thus it's unlikely that any widely available algorithm is going to be very useful out of the box unless you are doing something clever with it (at which point it sort of stops being widely available since you are adding to it).

That said, learning about autoregressive integerated moving average (ARIMA) models might be a useful start for forecasting time-series data. Don't expect better than random results though.

share|improve this answer
2  
+1: How many times I've been reading or been in a class and heard people who believe that if they have a complicated-enough algorithm, they could get rich in the stock/electricity/commodities markets. You try to explain overfitting, etc, but to no avail. Heck, as far as I know, not being an insider, successful stock-trading software has depended on no transaction fees, arbitrage, and high speed. The cutting edge now is to use loopholes in automated trading rules and high-speed proposal/withdrawl of bids to sucker-punch other automated traders. – Wayne Jan 21 '12 at 15:12
1  
The other issue is where multiple people end up with the same algorithm because they did their training on exactly the same data, then put volume sales/purchases through. Would any algorithm be expected to have long-run accuracy? – Michelle Jan 24 '12 at 19:33
@Wayne there are strategies for reducing over-fitting, though they're difficult to implement on time series data. – Zach Jan 27 '12 at 16:53
@Zach: yes, there are ways to penalize overfitting, but it's the attitude I'm reflecting on: the folks who have done some basic (probably erroneous) curve fitting in Excel and feel that they could've made money with their secret sauce, but what they really need is one of those cutting-edge, sophisticated algorithms that the professor just won't share with the class. That algorithm would fit the data like a glove, and then predict so much better than all those other speculators using Excel spreadsheets... but the professor keeps droning on about overfitting and the limitations of data. Sigh. – Wayne Jan 27 '12 at 19:32
up vote 7 down vote

I think for your purposes, you should pick a machine learning algorithm you find interesting and try it.

Regarding Efficient Market Theory, the markets are not efficient, in any time scale. Also, some people (both in academia and real-life quants) are motivated by the intellectual challenge, not just to get-rich-quick, and they do publish interesting results (and I count a failed result as an interesting one). But treat everything you read with a pinch of salt; if the results are really good, perhaps their scientific method isn't.

Data Mining With R might be a useful book for you; it is pricey, so try and find it in your university library. Chapter 2 covers just what you want to do, and he gets best results with a neural net. But be warned that he gets poor results, and spends a lot of CPU time to get them. The Amazon reviews point out the book costs $20 more because that chapter mentions the word finance; when reading it I got the impression the publisher had pushed him to write it. He's done his homework, read the docs, perused the right mailing lists, but his heart was not in it. I got some useful R knowledge from it, but won't be beating the market with it :-)

share|improve this answer
+1 That's a really thoughtful answer. – whuber♦ Jan 27 '12 at 17:34
@Darren - I like your style. – rolando2 Jan 28 '12 at 0:05
1  
A draft version (May 2003) of Data Mining with R can be found here. (I don't have the book, so I can't say what's the gap between the two versions.) – chl♦ Jan 29 '12 at 11:40
@chi Thanks! I took a quick look, and only two of the four chapters are there. But the bigger difference is that the Predicting Stock Market Returns chapter is very different. No mention of xts or quantmod and instead using the ts package, and using acf and the MARS package for predictions. It is almost like a bonus chapter, and I'm going to make time to read it properly. He is still using neural nets, but not comparing them to SVMs as in the published book. – Darren Cook Jan 30 '12 at 3:08
up vote 6 down vote

To my mind, any run-of-the-mill strong AI that could do all of the following might easily produce a statistically significant prediction:

  • Gather and understand rumours

  • Access and interpret all government knowledge

  • Do so in every relevant country

  • Make relevant predictions about:

    • Weather conditions

    • Terrorist activity

    • Thoughts and feelings of individuals

    • Everything else that affects trade

Statistical analysis is the least of your worries, really.

share|improve this answer
up vote 4 down vote

You could try the auto.arima and ets functions in R. You might also have some success with the rugarch package, but there's no existing functions for automated parameters selection. Maybe you could get parameters for the mean model from auto.arima, then pass them to rugarch and add garch(1,1)?

There's all sorts of blogs out there that claim some success doing this. Here's a system using an arima model (and later a garch model) and system using an SVM model. You'll find a lot of good info on FOSS trading, particularly if you start reading the blogs on his blogroll.

Whatever model you use, be sure to cross-validate and benchmark! I'd be very surprised if you found an arima, ets, or even garch model that could consistantly beat a naive model out-of-sample. Examples of time series cross-validation can be found here and here. Keep in mind that what you REALLY want to forecast is returns, not prices.

share|improve this answer
up vote 1 down vote

You should try GMDH-type neural networks. I know that some successful commercial packages for stock market prediction are using it, but mention it only in the depths of the documentation. In a nutshell it is a multilayered iterative neural network, so you are on the right way.

share|improve this answer
up vote 0 down vote

I think hidden markov models are popular in stock market. The most important thing to keep in mind is that you want an algorithm that preserves the temporal aspect of your data.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.