# Sequential experiment: controlled design and metrics

This is a generic question. Let me put forth an example scenario. Say, I had 2 techniques for allocating my daily budget within 4 stocks. Upon allocation, I get the data about the stock's performance the next day. (Am lazy, and go to a movie after allocation).

Problem 1:I have to evaluate or come up with an experimental design to test the two techniques in this sequential experiment over a given number of days.

Problem 2 with caveat: If I split my budget into half and have "one" of the two techniques allocate it amongst the four stocks in the same quantities in two parallel experiments at the same time-I find the following issue on the second day. Though everything from the budget to the allocation was the same. Both the experiments gave different performances -the next day- due to the randomness in the system. Under this situation- where performances are different even under "one" technique- How would I evaluate or design an experiment for comparing two techniques over a given number of days?

Problem 3: If instead of getting the performances the next day- I get minute by minute or hourly performances, and would like to evaluate the two technqiues-what would be your line of thought?

-
Hi there, to clarify, you wish to identify the best method to predict stock prices, the latter being a random/chaotic process? – Michelle Mar 9 '12 at 21:32
Hi-Without caveat: Would like to identify the best method, while the data comes in sequentially. aka- If I used a sequential hypothesis testing framework- I would like to account for testing the parameters from multiple distributions while accounting for nonlinearities (like in a generalized additive model) with distributional assumptions. – hotchpotch Mar 9 '12 at 22:06
Question with caveat: Do the same as above. But- when only one method is applied on the same four stocks by splitting the total money in half and allocating the same proportions on the same 4 stocks. i.e two replicates of same allocations on same stocks-The performances vary in both replicates. (varied initial conditions). Now how do you compare two methods sequentially in this scenario? – hotchpotch Mar 9 '12 at 22:08
The issue with trying to predict a random or chaotic series is that, basically, you can't. You may have some apparent short-run predictability by chance, but any model you come up with won't apply in the long-term. So by "stocks" do you mean shares (where my comments apply), or is this an inventory question (which could be predictable)? – Michelle Mar 9 '12 at 22:09
Fair enough- Lets exchange 'shares' with a more reasonable stocastic system- like inventory- or a more abstract form of resource allocation, and get back to the sequential hypothesis testing issue in this framework. Would like to do a sequential hypothesis testing- for two methods, where each point is modeled with a exponential distribution with different parameters like in GAM's and we have this setting for two methods. aka- sequential hypothesis testing at every point for two methods as data is collected and some form of an online likelihood ratio is measured. – hotchpotch Mar 9 '12 at 22:14

The question suggests that a repeated measures ANOVA could work, where technique defines the groups. For how frequently you should measure, that depends on what is important, i.e. the research hypothesis. If you're interested in minute outcomes, measure in minute lots. However, that will give you a lot of data points if you're doing this even over the space of a couple of hours, let alone days. If you end up with a lot of data points, the question is likely to be one of practical significance rather than statistical significance, as you are likely to end up with a statistically significant result regardless of how small the difference is between the techniques, just due to sheer volume of data.

-

The original post is unclear on what is the measure that you want to compare after the n days of the experiment.

The first alternative is that you want to decide which technique (A or B) will give you the best performance (on average) for the next day. That is, on average which of A and B will give you better returns FOR THE NEXT DAY.

A second alternative is which of A and B will give you (on average) the better return AFTER N DAYS. The two measures differ on the number of days you have to run the experiment to be sure of your answer.

a) Let us take the first metric - return for the next day.

You ran the experiment for N days, and at the beginning of the each day both A and B make their decision and you measure their performance at the end of the day. For each day both methods have the same information and the same budget, so you can pair each day of the experiment. So you would use a paired t-test (if N is large enough and the distribution of returns mildly normal- usually > 30) or a wilcoxon signed-rank test (if N is not that large or if the distributions are very non-normal).

If the resulting p-value is low enough, you will know which has better daily average return.

The only possible issue is that statistical tests assume that the samples are independently drawn from the population, but returns on following days are very likely not independent. So there is room for trouble here. But I believe that although stock prices or other time series data are very likely correlated form one time to the next one, from the experiment point of view, each return data is independent since at each day both A and B where allowed to make their choices. And you are measuring how good are the choices each method makes.

b) If you want the average return after N days, if you run the experiment for N days you only get a single data point for each method, and thus you do not have much evidence that one is better than the other. You would have to repeat this experiment M times (and calculate the N days average return for each method).

Again repeat the tests above for the set of M pairs of data, and you are done.

But there is a problem here - I believe you cannot overlap too much the intervals of N days of each experiment. On the limit, one could do the following. day 1 - A and B select stocks day 2 - same day 3 - same .... day N - same, but also measure the return for the N days starting at day 1 day N+1 - same, also measure the return for the N days starting at day 2 day N+2 - same, also measure the return for the N days starting at day 3

thus after N+M days you would have your M data pairs. But here the assumption of independence is clearly violated. The return for the first N days starting at day 1 is VERY correlated with the return of the N days starting at day 2, and so on. So the data is not interdependent, and thus the test is meaningless.

So in this case the safe thing to do is to use totally non-overlapping intervals of N days to make the measures.

-