# Experimental Design for Comparative Responses

Suppose I were looking to optimize the amount of certain spices in a chili spice recipe. The textbook experimental design would have me encode the amount of each spice in the design variable, choose some kind of design matrix, cook up a pot of chili for each row of the design matrix (one would have to take some shortcuts in the kitchen!), and assign a numeric dependent variable for each choice of the design variables. This latter operation is tricky for something like chili (or my real application, which is a secret); more likely a taste-tester could taste two different recipes and determine which is better, but assigning a numeric score is subject to all kinds of problems (drift over time, etc).

So I have a two part question, and they are intertwined:

1. Given the results of a number of such comparisons, how should one pick the optimal vector of design variables? (As a first pass, I might look into "Noisy Sorting", but something less fancy, based on logistic regression, say, might be a better choice.)
2. Once the evaluation method has been chosen, how should I design the experiment?
-
I don't understand how you can ask, "Given the results of a number of such comparisons, how should one pick the optimal vector of design variables?" and then "how should I design the experiment?" In other words, it sounds as if you want variable selection to take place both after and before you collect data. Am I missing something? – rolando2 Nov 22 '12 at 12:40
The goal is to find a good recipe; the variables have been preselected: $x_1$ is the amount of cayenne, $x_2$ is cumin, etc. By "pick the optimal vector of design variables," I meant, "infer, from all the data, the best recipe." That best recipe may be one that was tested, or it may be an interpolation of them. Sorry for the confusion. There is no variable selection. – shabbychef Nov 23 '12 at 4:58
Thanks. Sounds as if the big hurdle to get over (over which to get?) is determining a format for the dependent variable. You make a good point about drift over time potentially invalidating numeric ratings. – rolando2 Nov 23 '12 at 12:52

My short answer is have participants repeatedly taste and choose between recipes in a "2-interval forced choice paradigm" followed by logistic regression, using ingredients to predict probability of recipe choice.

In terms of the main experimental structure, we probably want each participant to do a number of two-interval forced-choice trials (2IFC, as they're called -- closely related to the 2AFC, or 2-alternative forced-choice) in which they taste two recipes in sequence (with an appropriate palette cleanser in between) and pick one. We want to give them all pairings of candidate recipes multiple times and balance the order of exposure within trials. In a perfect world, the experiment would be about as many minutes and spoonfuls as a meal, so that any effects of time (heat building or desensitization) remain with the normal range for a meal. But obviously there is a trade off with wanting to get more data.

Participants

Much of the difficulty with this sort of research is getting people to give good data. We probably need to screen out smokers, anyone recovering from a cold, and anyone else who might have decreased taste acuity. After we get our data, we will want to exclude particularly inconsistent participants using some predefined criterion. Similarly, people who show an unusual effect of within-trial stimulus order (preferring, say, the first over the second in each comparison) might also be given the hairy eyeball. Ideally we'd run a screening session at the start of the experiment and only advance people who give good data to our real tests.

There will still be potential issues with participants learning to discriminate or changing preference across trials (especially if the heat builds up or they desensitize to it). We may be able minimize this with our screening session or by using people with presumed stable preference as participants, such as experienced chili cooks. (Though we should ask ourselves what larger population we're trying to study. If we want to sell spice mixes maybe it is cooks; if we want to sell chili in a restaurant then we probably shouldn't just study cooks). No matter what, we will still probably want to exclude the first few trials from analysis as "practice trials" because they're always weird.

Analysis

For our main analysis, I might do a logistic regression predicting their probability of recipe preference using trial (grouping trials in some reasonable quantile like 4ths) and ingredient levels (and interactions) as fixed effects.

One can imagine also needing a total heat random effect that might vary by participant. Or we could determine each participants' preferred Scoville units in screening and keep it constant.

Generally, the relationship between a physical stimulus and its perceived intensity isn't linear and is often said to be exponential (Stevens power law). So we may end up transforming our ingredient amount variables.

Last Thoughts

Finally, experience tells us that we seldom run a first experiment that perfectly answers the right question. And it may turn out that the real question isn't preferred spicing of chili in the context of other chili recipes, but preference in the context of a meal. And our design is unable to assess most potential adverse effects. And we're not dealing with the fact that natural products vary in their chemical constituents and flavors. But you have to start somewhere.