Sequential experiment: controlled design and metrics This is a generic question. Let me put forth an example scenario. Say, I had 2 techniques for allocating my daily budget within 4 stocks. Upon allocation, I get the data about the stock's performance the next day. (Am lazy, and go to a movie after allocation).
Problem 1:I have to evaluate or come up with an experimental design to test the two techniques in this sequential experiment over a given number of days. 
Problem 2 with caveat: If I split my budget into half and have "one" of the two techniques allocate it amongst the four stocks in the same quantities in two parallel experiments at the same time-I find the following issue on the second day. Though everything from the budget to the allocation was the same. Both the experiments gave different performances -the next day- due to the randomness in the system. Under this situation- where performances are different even under "one" technique- How would I evaluate or design an experiment for  comparing two techniques over a given number of days?
Problem 3: If instead of getting the performances the next day- I get minute by minute or hourly performances, and would like to evaluate the two technqiues-what would be your line of thought? 
 A: The question suggests that a repeated measures ANOVA could work, where technique defines the groups. For how frequently you should measure, that depends on what is important, i.e. the research hypothesis. If you're interested in minute outcomes, measure in minute lots. However, that will give you a lot of data points if you're doing this even over the space of a couple of hours, let alone days. If you end up with a lot of data points, the question is likely to be one of practical significance rather than statistical significance, as you are likely to end up with a statistically significant result regardless of how small the difference is between the techniques, just due to sheer volume of data.
A: The original post is unclear on what is the measure that you want to compare after the n days of the experiment. 
The first alternative is that you want to decide which technique (A or B) will give you the best performance (on average) for the next day. That is, on average which of A and B will give you better returns FOR THE NEXT DAY. 
A second alternative is which of A and B will give you (on average) the better return AFTER N DAYS. The two measures differ on the number of days you have to run the experiment to be sure of your answer.
a) Let us take the first metric -  return for the next day. 
You ran the experiment for N days, and at the beginning of the each day both A and B make their decision and you measure their performance at the end of the day. For each day both methods have the same information and the same budget, so you can pair each day of the experiment. So you would use a paired t-test (if N is large enough and the distribution of returns mildly normal- usually > 30) or a wilcoxon signed-rank test  (if N is not that large or if the distributions are very non-normal).
If the resulting p-value is low enough, you will know which has better daily average return.
The only possible issue is that statistical tests assume that the samples are independently drawn from the population, but returns on following days are very likely not independent. So there is room for trouble here. But I believe that although stock prices or other time series data are very likely correlated form one time to the next one, from the experiment point of view, each return data is independent since at each day both A and B where allowed to make their choices. And you are measuring how good are the choices each method makes. 
b) If you want the average return after N days, if you run the experiment for N days you only get a single data point for each method, and thus you do not have much evidence that one is better than the other. You would have to repeat this experiment M times (and calculate the N days average return for each method). 
Again repeat the tests above for the set of M pairs of data, and you are done. 
But there is a problem here - I believe you cannot overlap too much the intervals of N days of each experiment. On the limit, one could do the following.
day 1 - A and B select stocks
day 2 - same
day 3 - same
....
day N - same, but also measure the  return for the N days starting at day 1
day N+1 - same, also measure the  return for the N days starting at day 2
day N+2 - same, also measure the  return for the N days starting at day 3
thus after N+M days you would have your M data pairs. But here the assumption of independence is clearly violated. The return for the first N days starting at day 1 is VERY correlated with the return of the N days starting at day 2, and so on. So the data is not interdependent, and thus the test is meaningless. 
So in this case the safe thing to do is to use totally non-overlapping intervals of N days to make the measures.
