Experimental Design for Comparative Responses Suppose I were looking to optimize the amount of certain spices in a chili spice recipe. The textbook experimental design would have me encode the amount of each spice in the design variable, choose some kind of design matrix, cook up a pot of chili for each row of the design matrix (one would have to take some shortcuts in the kitchen!), and assign a numeric dependent variable for each choice of the design variables. This latter operation is tricky for something like chili (or my real application, which is a secret); more likely a taste-tester could taste two different recipes and determine which is better, but assigning a numeric score is subject to all kinds of problems (drift over time, etc). 
So I have a two part question, and they are intertwined:


*

*Given the results of a number of such comparisons, how should one pick the optimal vector of design variables? (As a first pass, I might look into "Noisy Sorting", but something less fancy, based on logistic regression, say, might be a better choice.)

*Once the evaluation method has been chosen, how should I design the experiment? 

 A: My short answer is have participants repeatedly taste and choose between recipes in a "2-interval forced choice paradigm" followed by logistic regression, using ingredients to predict probability of recipe choice.  
Long Answer:
In terms of the main experimental structure, we probably want each participant to do a number of two-interval forced-choice trials (2IFC, as they're called -- closely related to the 2AFC, or 2-alternative forced-choice) in which they taste two recipes in sequence (with an appropriate palette cleanser in between) and pick one. We want to give them all pairings of candidate recipes multiple times and balance the order of exposure within trials. In a perfect world, the experiment would be about as many minutes and spoonfuls as a meal, so that any effects of time (heat building or desensitization) remain with the normal range for a meal.  But obviously there is a trade off with wanting to get more data.
Participants
Much of the difficulty with this sort of research is getting people to give good data. We probably need to screen out smokers, anyone recovering from a cold, and anyone else who might have decreased taste acuity.  After we get our data, we will want to exclude particularly inconsistent participants using some predefined criterion. Similarly, people who show an unusual effect of within-trial stimulus order (preferring, say, the first over the second in each comparison) might also be given the hairy eyeball. Ideally we'd run a screening session at the start of the experiment and only advance people who give good data to our real tests. 
There will still be potential issues with participants learning to discriminate or changing preference across trials (especially if the heat builds up or they desensitize to it).  We may be able minimize this with our screening session or by using people with presumed stable preference as participants, such as experienced chili cooks. (Though we should ask ourselves what larger population we're trying to study.  If we want to sell spice mixes maybe it is cooks; if we want to sell chili in a restaurant then we probably shouldn't just study cooks).  No matter what, we will still probably want to exclude the first few trials from analysis as "practice trials" because they're always weird. 
Analysis
For our main analysis, I might do a logistic regression predicting their probability of recipe preference using trial (grouping trials in some reasonable quantile like 4ths) and ingredient levels (and interactions) as fixed effects. 
One can imagine also needing a total heat random effect that might vary by participant.  Or we could determine each participants' preferred Scoville units in screening and keep it constant. 
Generally, the relationship between a physical stimulus and its perceived intensity isn't linear and is often said to be exponential (Stevens power law).  So we may end up transforming our ingredient amount variables. 
Last Thoughts
Finally, experience tells us that we seldom run a first experiment that perfectly answers the right question.  And it may turn out that the real question isn't preferred spicing of chili in the context of other chili recipes, but preference in the context of a meal.  And our design is unable to assess most potential adverse effects.  And we're not dealing with the fact that natural products vary in their chemical constituents and flavors.  But you have to start somewhere. 
Fun reading
Bartoshuk, Linda M. "Comparing sensory experiences across individuals: recent psychophysical advances illuminate genetic variation in taste perception." Chemical Senses 25.4 (2000): 447-460.
Chapman, K. W., K. GRACE‐MARTIN, and H. T. Lawless. "Expectations and stability of preference choice." Journal of sensory studies 21.4 (2006): 441-455.
Karrer, T., and L. Bartoshuk. "Capsaicin desensitization and recovery on the human tongue." Physiology & behavior 49.4 (1991): 757.
Lawless, Harry T., and Hildegarde Heymann. "Physiological and psychological foundations of sensory function." Sensory Evaluation of Food (2010): 19-56.
