General theories would be great. My specific problem is I'm trying to find a specific portfolio of all the stocks in the market. The possibilities are huge because I need the stock combination(itself massive) and weights for each stock(even larger than the quadrillion combination). To make matters more complicated, I'm taking daily data but am trying to build a long term model.
Here's an example of what I'm trying to do(its impossible because of the size but just so it makes it more clear), I take all the possible portfolio that have a return of more than 3% today(lets pretend I need to study this on the portfolio as a whole and it cannot be derived from individual stocks). Then, I look at that portfolio and see if it did the same thing yesterday, day before that and so on..after going back X days, then I return the portfolio.
This way is not feasible because the entire population is huge(it would take a few billion years to process all combinations) and the results that work everyday are huge as well. If I'm trying to build portfolios how can I intelligently draw samples from this? I've tried genetic algorithms to mutate successful ones but the problem is it may work but I might be in a local peak, so I tried to do random sampling as well but that didn't work very well. I'm trying to think of ways of not sampling randomly. Any ideas?
(if its not clear here's another problem that is more general but also illustrates where I'm lost when it comes to sampling): say you wanted to know what percent of the entire world liked Democrats every year. The world is too large to poll everyone and its very diverse so doing it randomly may give you areas with high population(i.e. india/China but you'd miss out on smaller countries). Plus to complicate things, the answer this year will cause you to go back to look at the answer they gave last year and so on. I'm interested in the sampling techniques, do you somehow segment the world into smaller groups and equally take samples(then a small country would have just as many votes as a large one? or do you do it randomly? etc..).
p.s. I know this is highly dependent on my research set but I'm trying to learn about general ideas so I can use existing methods as a model of thinking about this problem. I'm have taken basic stats courses back in school but I'm not sure if this question is about sampling or if its about random sampling. Any suggestions are welcomed.