My girlfriend has recently gotten a job doing sales and trading at a major bank. Buoyed by her new job, she believes she can predict whether stocks will be up or down at the end of the month greater than chance (she believes she can even do it with 80% accuracy!)

I am very skeptical. We have agreed to do an experiment in which she will choose a number of stocks and, at a predetermined time, we will check if they are up or down.

My question is this: how many stocks would she have to pick, and how many would she have to get right, in order to have enough statistical power to tell with confidence that she can accurately predict stocks?

For example, how many stocks would she have to pick to tell with 95% certainty that she pick stocks with 80% accuracy?

Edit: For the experiment we agreed to, she does not have to predict by how much stocks will be up or down, but only if they will be up or down.

  • 1
    $\begingroup$ Sort of an unusual question for the site, but interesting I think. One interesting aspect of this question is what kind of statistical model we think reasonably represents a baseline / non-psychic forecaster. Like, if a particular stock has risen by +3 every day for the last month, you wouldn't need to be psychic to reasonably suppose that it may be up +3 again tomorrow. So how accurate does she need to be (and compared to what) before we accept it as evidence of precognition? It's not immediately clear to me. $\endgroup$ Commented May 1, 2018 at 21:55
  • 3
    $\begingroup$ Do her predictions also make her profit? That's what matters. The mere predictions of rise and fall (unweighted) are easy otherwise. $\endgroup$ Commented May 1, 2018 at 22:00
  • 5
    $\begingroup$ Stocks in a bull market will go up anyway. You could compare with a market index (can she outperform just buying the market?). What would justify her feeling like she can "pick" stocks - if she would be picking ones that do better than that (otherwise why seek investment advice at all? Just buy the market). I'd suggest actually using her picks as a portfolio (weighted how she likes) and see how that performs. I also wouldn't do it once, but many times. In addition, you have to consider the cost of following her advice (how often would she get people to change what they hold?). ... ctd $\endgroup$
    – Glen_b
    Commented May 1, 2018 at 22:49
  • 3
    $\begingroup$ ... Trades cost money, so even if she outperforms the market, does she still do it after you adjust for the cost of actually doing the trades? Now, presumably she would be paid for her advice. Does she bring enough extra value above what she charges that they still outperform the market after paying her (or her bank's) fees and the trading costs? [If not, her advice isn't actually worth anything to a client.] .... studies tend to show that knowledgeable advisers can outperform the market on average (if only just) but once you take the various costs into account, they don't. $\endgroup$
    – Glen_b
    Commented May 1, 2018 at 22:56
  • 4
    $\begingroup$ Consider whether it's a good idea to measure the evidence that your girlfriend is wrong. $\endgroup$
    – Quasar
    Commented May 2, 2018 at 20:32

3 Answers 3


Interesting question. This isn’t really an answer, but it’s too long to be a comment.

I think your experimental design is challenged for these reasons:

1) This does not reflect the way that stock picking is actually evaluated in the “real world”. As an extreme example, suppose stock picker A chose 1 stock that went up 1000%, and 9 that went down by 1%, and stock picker B chose 10 stocks that all went up 1%. If these stocks were actually used to construct an index, then clearly A would be the better performer, but B would do much better in your experiment. A more financially interesting challenge would be to construct a portfolio and compare its performance to that of the S&P 500. In turn, there is a commonly-used machinery for evaluating such performance: simply take a linear regression of the day-to-day returns of the portfolio against those of the S&P. The intercept term (often called “alpha”) measures the average performance “over and above the market”. Since it is a coefficient of a linear regression, it is a trivial matter to construct a 95% confidence interval if you so choose. Then compare this to the fees her bank would charge for this service.

2) Disregarding 1, since it sounds like you both have already agreed on the form the experiment, consider how this could be gamed. Suppose I had a magic oracle that told me the probability of each stock being above its current price a month from now (say). Then I could just pick the n stocks with the highest such probabilities, and most likely over 50% of them would indeed go up. Now, such probabilities are encoded (imperfectly) in various options prices. For example, I can buy a so-called “binary option”, which is basically just a gamble on the event “Stock X willl be above price Y on date Z”. The pricing of such implies a probability of this event (although the closer date Z is to the present, the less reliable this will be). Since blindly following the “wisdom of the crowds” requires no particular expertise, I would argue that the performance of a strategy like this should be considered “chance levels” for your particular experiment. Alternatively, you present her with a list of stocks of your choosing, and have her indicate whether she thinks each will be up or down, together with her confidence on each prediction. Then group all answers by confidence level and see how closely they align (i.e., of those stocks that she was 90% confident about, did she correctly predict 90% of them?). There’s a standard way to quantify this; i don’t remember offhand what it’s called, but you can read about it in Superforecasters by Phil Tetlock.

  • 1
    $\begingroup$ Long comments can be constructed using multiple comments. The answer space is meant t be used only for answers. $\endgroup$ Commented May 2, 2018 at 18:40
  • 2
    $\begingroup$ It looks like at least a partial answer. If anything more comments like this should probably be answers (my own included). $\endgroup$
    – Glen_b
    Commented May 2, 2018 at 23:45

A very simple test would be as follows: Whenever she picks a stock, you pick one stock as well. I reckon you don't think of yourself as been an expert in the stock market. Hence, your choice will be approx. random.

Using this method, you can improve the statistical power by imposing some rules:

  1. Both of you assign the same forecast (decrease or increase). She is allowed to choose which one.
  2. You should define at what time you evaluate the stocks.
  3. You should define how many stocks you have to buy (>20 would be nice) and that you have to buy them for the same amount of money. Hence, when she says she buys stock A, that implies that she will buy them for 10 000 dollars.
  4. Things become more precise, if both of you limit your choices to stocks of a special index. Than you don't have to pick any stocks, but you could run a simulation. Then you could even evaluate the expected variance. However, you will need do store the stock data somewhere. An alternative would be, that when ever she buy a stock, you pick 10 random stocks -- you just simulate the pick of ten random "experts". :)

How much power do you want your statistical test to have? That is, if she does have the ability, with what probability do you want to detect the ability? Defining power is essential to determining sample size.

To provide an answer, let's make some assumptions

  1. Let's assume we want a power of 80%, and confidence level of 95%, and a one sided test.
  2. To prevent making a single prediction (i.e. everything stock will go up), force her to predict n markets that will go up and n markets that will go down. This will ensure that she can predict the ones that will go up as well as the ones that will go down.
  3. We will test against a random guesser (50:50), i.e. $H_0: p>0.5$.

Under this frame work, she would have to pick 15 stocks that will go up, and 15 stocks that will go down.

Link to calculator

  • $\begingroup$ 1) I guess that you will have to test against p=0.8. 2) also, it might be better to have her guess for an x number of random stocks after all. Because in this concept with stocks of her own choice she might choose those stocks which are easiest to predict. $\endgroup$ Commented May 2, 2018 at 21:29
  • $\begingroup$ In this alternative test (random stocks), to test she has power at least 80% a test with x=14 would be sufficient (where she has to guess all of them correct). If she has really 0.8 or higher power then the probability to get not any error is less than 5% (or more exactly pbinom(0,14,0.2)=0.044) this resembles Fisher's tea tasting experiment $\endgroup$ Commented May 2, 2018 at 21:32
  • $\begingroup$ I think the concept of her choosing 15 (or so) on her own is a more representative way of how one might choose stocks. If she can reliably choose 15 stock that will go up (p>.5) and ones that will go down (p>.5) then money can most likely be made. She gets to pick the stocks she is most confident about in this test and in her job (she should choose the "easy to predict stocks") $\endgroup$
    – Underminer
    Commented May 2, 2018 at 21:49
  • $\begingroup$ That is indeed a more representative way of what one would do in a job. But the 'predict whether stocks will be up or down with .8 accuracy' is different from 'select a few stocks that will be up or down with .8 accuracy'. Then this accuracy level will also depend on the number of stocks she has to choose. It would become more difficult if you ask to select something >15 stocks. Then the issue 'to choose the minimal number of stocks that need to be selected' is not like the problem of the tea tasting experiment, but about what would be a representative number for the job (15 might be too easy) $\endgroup$ Commented May 2, 2018 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.