This is a generic question. Let me put forth an example scenario. Say, I had 2 techniques for allocating my daily budget within 4 stocks. Upon allocation, I get the data about the stock's performance the next day. (Am lazy, and go to a movie after allocation).
Problem 1:I have to evaluate or come up with an experimental design to test the two techniques in this sequential experiment over a given number of days.
Problem 2 with caveat: If I split my budget into half and have "one" of the two techniques allocate it amongst the four stocks in the same quantities in two parallel experiments at the same time-I find the following issue on the second day. Though everything from the budget to the allocation was the same. Both the experiments gave different performances -the next day- due to the randomness in the system. Under this situation- where performances are different even under "one" technique- How would I evaluate or design an experiment for comparing two techniques over a given number of days?
Problem 3: If instead of getting the performances the next day- I get minute by minute or hourly performances, and would like to evaluate the two technqiues-what would be your line of thought?