Multivariate Hypothesis Testing - How do you choose best combination of factors without explicitly testing every combination (2^n)? Let's say I have 10 variables that I want to vary (each can be 0/1) on a page in order to determine what leads to the best conversion. How can I choose the best combination of factors without having 2^10 separate splits?
 A: This is actually a fairly difficult problem to answer, there are no fully agreed upon methods to to partial-factorial testing online (you don't want to do full-factorial, as you mentioned). Taguchi methods more info here https://en.wikipedia.org/wiki/Taguchi_methods] are popular in manufacturing to achieve this type of testing where you infer what would happen with certain interactions that you do not actually test. The challenge in the online environment is that we are not dealing with things that have immutable qualities or known interactions (e.g. if I test two things that are steel I can be reasonably confident how they will interact, whereas if I change a website's design it's hard to guess how humans will change their feelings about it) so while it it possible (for example the excellent tool Maxymizer offers Taguchi as an alternative to A/B/N testing) there are varying feeling about its effectiveness.
That said, if you are mentioning 10 different factors, some are likely very small factors, in which case you would be pretty safe assuming certain interactions won't have huge difference if they are small and not on the same feature.
The easiest method is what I would recommend, and what we did in many cases running huge multivariate tests in online advertising, is to run a few variations you feel good about, establish the victorious variation (bonus points for using multi-armed bandit to allocate traffic towards the best performing option on the way, for a great treatment on the topic read this: https://support.google.com/analytics/answer/2844870?hl=en). Next test new options against the current winner, you will pick up what is working more quickly and dilute your traffic a bit less while running something that is working reasonably well, so you will get to a decent place and not be languishing in trying to get tons of traffic for your test to be done.
You will also be able to see which factors have very little effect on the overall outcome. If you see that changing factor 3 never has much effect you can start removing things you can vary - frankly 10 items is a lot and you can probably do some feature reduction which will also improve your ability to get this done.
