How to tell if girlfriend can tell the future (i.e. predict stocks)?

Question

My girlfriend has recently gotten a job doing sales and trading at a major bank. Buoyed by her new job, she believes she can predict whether stocks will be up or down at the end of the month greater than chance (she believes she can even do it with 80% accuracy!)

I am very skeptical. We have agreed to do an experiment in which she will choose a number of stocks and, at a predetermined time, we will check if they are up or down.

My question is this: how many stocks would she have to pick, and how many would she have to get right, in order to have enough statistical power to tell with confidence that she can accurately predict stocks?

For example, how many stocks would she have to pick to tell with 95% certainty that she pick stocks with 80% accuracy?

Edit: For the experiment we agreed to, she does not have to predict by how much stocks will be up or down, but only if they will be up or down.

Answer 1

Interesting question. This isn’t really an answer, but it’s too long to be a comment.

I think your experimental design is challenged for these reasons:

1) This does not reflect the way that stock picking is actually evaluated in the “real world”. As an extreme example, suppose stock picker A chose 1 stock that went up 1000%, and 9 that went down by 1%, and stock picker B chose 10 stocks that all went up 1%. If these stocks were actually used to construct an index, then clearly A would be the better performer, but B would do much better in your experiment. A more financially interesting challenge would be to construct a portfolio and compare its performance to that of the S&P 500. In turn, there is a commonly-used machinery for evaluating such performance: simply take a linear regression of the day-to-day returns of the portfolio against those of the S&P. The intercept term (often called “alpha”) measures the average performance “over and above the market”. Since it is a coefficient of a linear regression, it is a trivial matter to construct a 95% confidence interval if you so choose. Then compare this to the fees her bank would charge for this service.

2) Disregarding 1, since it sounds like you both have already agreed on the form the experiment, consider how this could be gamed. Suppose I had a magic oracle that told me the probability of each stock being above its current price a month from now (say). Then I could just pick the n stocks with the highest such probabilities, and most likely over 50% of them would indeed go up. Now, such probabilities are encoded (imperfectly) in various options prices. For example, I can buy a so-called “binary option”, which is basically just a gamble on the event “Stock X willl be above price Y on date Z”. The pricing of such implies a probability of this event (although the closer date Z is to the present, the less reliable this will be). Since blindly following the “wisdom of the crowds” requires no particular expertise, I would argue that the performance of a strategy like this should be considered “chance levels” for your particular experiment. Alternatively, you present her with a list of stocks of your choosing, and have her indicate whether she thinks each will be up or down, together with her confidence on each prediction. Then group all answers by confidence level and see how closely they align (i.e., of those stocks that she was 90% confident about, did she correctly predict 90% of them?). There’s a standard way to quantify this; i don’t remember offhand what it’s called, but you can read about it in Superforecasters by Phil Tetlock.

Answer 2

A very simple test would be as follows: Whenever she picks a stock, you pick one stock as well. I reckon you don't think of yourself as been an expert in the stock market. Hence, your choice will be approx. random.

Using this method, you can improve the statistical power by imposing some rules:

1. Both of you assign the same forecast (decrease or increase). She is allowed to choose which one.
2. You should define at what time you evaluate the stocks.
3. You should define how many stocks you have to buy (>20 would be nice) and that you have to buy them for the same amount of money. Hence, when she says she buys stock A, that implies that she will buy them for 10 000 dollars.
4. Things become more precise, if both of you limit your choices to stocks of a special index. Than you don't have to pick any stocks, but you could run a simulation. Then you could even evaluate the expected variance. However, you will need do store the stock data somewhere. An alternative would be, that when ever she buy a stock, you pick 10 random stocks -- you just simulate the pick of ten random "experts". :)

Answer 3

How much power do you want your statistical test to have? That is, if she does have the ability, with what probability do you want to detect the ability? Defining power is essential to determining sample size.

To provide an answer, let's make some assumptions

1. Let's assume we want a power of 80%, and confidence level of 95%, and a one sided test.
2. To prevent making a single prediction (i.e. everything stock will go up), force her to predict n markets that will go up and n markets that will go down. This will ensure that she can predict the ones that will go up as well as the ones that will go down.
3. We will test against a random guesser (50:50), i.e. $H_0: p>0.5$.

Under this frame work, she would have to pick 15 stocks that will go up, and 15 stocks that will go down.

Link to calculator