# How good is an optimal design (DOE)?

I am looking at an experiment with 5 factors (4 numeric and one nominal) with three levels for each numeric and 2 levels for the nominal. Instead of the 162 runs, I am interested in a small design (30 runs) using a specific postulated model for the response surface. Using the AlgDesign library in R I am able to generate these 30 design points:

library(AlgDesign)

candidates <- gen.factorial(levels = c(3,3,3,3,2),
#code all as numeric,  even factors so can do interactions
factors = NULL,
varNames = c("intro","duration","goto","fee","color")
)
candidates

desD <- optFederov(frml = ~intro+goto+duration+fee+color
+ I(intro*intro)
+I(goto*goto)
+I(duration*duration)
+I(fee*fee)
+ I(intro*goto)
+ I(intro*duration)
+ I(intro*fee)
+ I(intro*color)
+ I(goto*duration)
+ I(goto*fee)
+ I(goto*color)
+ I(duration*fee)
+ I(duration*color)
+ I(fee*color),
data = candidates,
nTrials=30,
criterion = "D",
maxIteration = 1000,
eval=TRUE,
nRepeats = 10)


Great, so now how good is this? The D criteria is

> desD$D [1] 0.5422988  Question #1: Is there a rule of thumb for if this is "acceptable"? The library allows one to run the design through an evaluation. eval.design(frml = ~intro+goto+duration+fee+color + I(intro*intro) +I(goto*goto) +I(duration*duration) +I(fee*fee) + I(intro*goto) + I(intro*duration) + I(intro*fee) + I(intro*color) + I(goto*duration) + I(goto*fee) + I(goto*color) + I(duration*fee) + I(duration*color) + I(fee*color),design = desD$design,confounding = TRUE,X = candidates)


Which produces

$confounding [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] (Intercept) -1.0000 -0.0898 0.0726 -0.4094 0.0383 -0.3290 0.7405 0.7806 0.7324 0.7541 0.2098 intro -0.0062 -1.0000 0.0842 0.0805 -0.0332 0.0354 -0.0087 -0.0645 0.0805 -0.0050 0.0038 goto 0.0049 0.0810 -1.0000 0.0060 0.0130 -0.0041 -0.0118 0.0066 -0.0063 0.0001 -0.0849 duration -0.0278 0.0787 0.0061 -1.0000 -0.0250 0.0203 0.0770 0.0297 -0.0185 -0.0076 0.0246 fee 0.0025 -0.0311 0.0126 -0.0239 -1.0000 -0.0125 0.0084 -0.0262 0.0077 0.0004 0.0568 color -0.0177 0.0273 -0.0033 0.0161 -0.0103 -1.0000 -0.0020 0.0258 -0.0001 0.0031 0.0097 I(intro * intro) 0.3066 -0.0517 -0.0728 0.4690 0.0536 -0.0151 -1.0000 -0.0058 0.0647 0.0602 0.0813 I(goto * goto) 0.2605 -0.3092 0.0330 0.1456 -0.1339 0.1602 -0.0047 -1.0000 0.0265 0.0040 0.0549 I(duration * duration) 0.2905 0.4590 -0.0373 -0.1078 0.0471 -0.0007 0.0620 0.0315 -1.0000 0.0339 -0.1652 I(fee * fee) 0.2847 -0.0271 0.0005 -0.0423 0.0026 0.0217 0.0549 0.0045 0.0323 -1.0000 -0.1389 I(intro * goto) 0.0169 0.0044 -0.1024 0.0292 0.0703 0.0145 0.0159 0.0133 -0.0336 -0.0297 -1.0000 I(intro * duration) 0.0091 0.0349 0.0368 0.0470 -0.1013 0.0590 0.0002 0.0080 0.0105 -0.0379 0.0443 I(intro * fee) 0.0189 0.0570 0.0485 -0.1040 -0.0341 0.0382 0.0154 -0.0077 -0.0630 -0.0096 -0.0486 I(intro * color) 0.0034 -0.2138 0.0249 0.0622 0.0264 -0.0187 0.0119 -0.0429 0.0152 0.0019 -0.0801 I(goto * duration) -0.0004 0.0445 0.0078 -0.0399 -0.0384 -0.0062 0.0505 -0.0012 -0.0039 -0.0493 0.0320 I(goto * fee) -0.0003 0.0332 -0.0133 -0.0099 -0.0060 0.0044 -0.0462 0.0064 0.0496 -0.0052 -0.0274 I(goto * color) 0.0259 0.0300 -0.1740 0.0400 0.0035 -0.0015 -0.0885 0.0056 0.0061 0.0042 0.0055 I(duration * fee) -0.0054 -0.0940 -0.0304 0.0287 -0.0454 0.0472 0.0381 -0.0191 0.0003 -0.0130 -0.0549 I(duration * color) -0.0016 0.0501 -0.0085 -0.1986 0.0270 -0.0226 0.0242 -0.0199 -0.0049 0.0047 -0.0344 I(fee * color) -0.0033 0.0341 0.0013 0.0257 -0.1876 0.0169 -0.0100 0.0256 -0.0033 -0.0013 0.0476 [,12] [,13] [,14] [,15] [,16] [,17] [,18] [,19] [,20] (Intercept) 0.1190 0.2467 0.0508 -0.0053 -0.0038 0.3719 -0.0706 -0.0239 -0.0508 intro 0.0318 0.0518 -0.2232 0.0389 0.0290 0.0300 -0.0859 0.0536 0.0367 goto 0.0322 0.0425 0.0250 0.0066 -0.0112 -0.1672 -0.0267 -0.0088 0.0013 duration 0.0418 -0.0925 0.0635 -0.0341 -0.0085 0.0391 0.0256 -0.2079 0.0271 fee -0.0865 -0.0291 0.0259 -0.0314 -0.0049 0.0033 -0.0389 0.0271 -0.1893 color 0.0415 0.0269 -0.0151 -0.0042 0.0030 -0.0011 0.0333 -0.0187 0.0141 I(intro * intro) 0.0009 0.0831 0.0741 0.2624 -0.2401 -0.5263 0.2069 0.1540 -0.0642 I(goto * goto) 0.0350 -0.0334 -0.2150 -0.0049 0.0269 0.0270 -0.0838 -0.1023 0.1324 I(duration * duration) 0.0543 -0.3265 0.0902 -0.0192 0.2471 0.0349 0.0013 -0.0298 -0.0205 I(fee * fee) -0.1875 -0.0474 0.0107 -0.2338 -0.0245 0.0230 -0.0643 0.0276 -0.0074 I(intro * goto) 0.0468 -0.0512 -0.0970 0.0324 -0.0278 0.0063 -0.0582 -0.0427 0.0595 I(intro * duration) -1.0000 0.0108 0.0672 0.0370 -0.0563 -0.0412 -0.0954 0.0589 -0.0214 I(intro * fee) 0.0108 -1.0000 -0.0388 -0.0566 0.0585 0.0647 0.0000 -0.0220 0.0418 I(intro * color) 0.0587 -0.0338 -1.0000 -0.0307 0.0495 0.0947 -0.0176 0.0419 -0.0098 I(goto * duration) 0.0385 -0.0590 -0.0367 -1.0000 -0.0296 0.0317 0.0420 -0.0029 0.0393 I(goto * fee) -0.0586 0.0609 0.0591 -0.0296 -1.0000 -0.0335 -0.0375 0.0442 0.0025 I(goto * color) -0.0376 0.0589 0.0989 0.0278 -0.0293 -1.0000 0.0519 0.0125 -0.0087 I(duration * fee) -0.0951 0.0000 -0.0201 0.0401 -0.0359 0.0568 -1.0000 -0.0309 0.0414 I(duration * color) 0.0501 -0.0187 0.0408 -0.0023 0.0361 0.0117 -0.0264 -1.0000 -0.0170 I(fee * color) -0.0181 0.0353 -0.0095 0.0319 0.0020 -0.0080 0.0351 -0.0169 -1.0000$determinant
[1] 0.5422988

$A [1] 3.51777$I
[1] 20.58251

$Geff [1] 0.696$Deffbound
[1] 0.646

$diagonality [1] 0.782$gmean.variances
[1] 1.966862


Question #2: From one of the package vignettes , this is a statement made about the design they were observing and the eval function. Does one look then at these things only from a relative perspective (unlike fractional factorials where effects are either clear or confounded) where diagonality of 0.78 is pretty good (since 1 is perfect) and if there are entries in the confounding matrix that are "large" then we consider the effects as problematic to clearly estimate them?

EDIT: 1

Here is one thought I had - perhaps someone could give their thoughts if this approach is valid to make sure the effects are somewhat non-confounded.

Create dummy response data, fit the model of interest and then check VIF. Here, none are above 2 so we feel good about being able to get good clear estimates of the effects.

    #dummy response
y <- rbinom(nrow(desD$design),size = 12000,prob = 0.009) non_response<-12000-y mod <- glm(cbind(y,non_response)~intro+goto+duration+fee+color + I(intro*intro) +I(goto*goto) +I(duration*duration) +I(fee*fee) + I(intro*goto) + I(intro*duration) + I(intro*fee) + I(intro*color) + I(goto*duration) + I(goto*fee) + I(goto*color) + I(duration*fee) + I(duration*color) + I(fee*color), data=desD$design, family = "binomial")

library(car)

car::vif(mod)

intro                   goto               duration                    fee                  color
1.045220               1.066770               1.095618               1.159182               1.083592
I(intro * intro)         I(goto * goto) I(duration * duration)           I(fee * fee)        I(intro * goto)
1.136307               1.225223               1.238652               1.104796               1.053494
I(intro * duration)         I(intro * fee)       I(intro * color)     I(goto * duration)          I(goto * fee)
1.061060               1.038913               1.144018               1.058265               1.044455
I(goto * color)      I(duration * fee)    I(duration * color)         I(fee * color)
1.050094               1.063997               1.066333               1.100737

• I have always thought of "optimal" as either true or false. This comes from "control system engineering" first and "doe" second. In DOE you can constrain your domain to integer values or such, so that extremizing the measure of goodness is not as extremal if continuous values were allowed, but still it qualifies as optimal. Personally, I would not like d-optimal that wasn't actually d-optimal or optimal. – EngrStudent Dec 18 '16 at 3:12
• There is more than one criteria for optimal design. In the realm of DOE we also have the concept of robust design that Taguchi introduced. People like George Box and Jeff Wu followed up on refining it. Maybe you should look at Jeff's book. – Michael Chernick Dec 18 '16 at 16:49
• A URL for Jeff's book :www.wiley.com/WileyCDA/WileyTitle/productCd-0471699462.html . – Michael Chernick Dec 18 '16 at 17:01
• First you must define what "good" means to you, operationally. Then simulate data many times from the model, using your design, evaluate your criteria, and look at a histogram of criterion values. Then you can tell if this design is "good" for you! – kjetil b halvorsen Dec 31 '16 at 15:57
• There's no rule of thumb that applies to all experiments. Like, if you're working at CERN looking for the Higgs boson, you'll want astronomical amounts of data before even considering a result to be potentially valid; you can't just walk in there and tell them that 30 points, or 162, are enough - that's silly. So, if you're looking for some set approach to profiling a physical system's response in a limited set of evaluations, presumably you'll want to have some prior knowledge of that system and a reason to believe that it's sufficiently well-behaved. – Nat Jan 1 '17 at 10:24