7
$\begingroup$

I would like to perform some black box testing on some software that takes a large number of input variables. These variables can have some interactions (either intentional or non-intentional) so I would like to test a large number of combinations of them and then compare them to the output of another piece of software that is accepted as truth.

Right now I have about 7 3-level variables and about 8 2-level variables, so we talking about half a million runs to do a full factorial test. The software is a little slow, so I'd really like to keep it around the ball park of a few thousand runs.

I'm by no means a statistician and only first heard about design of experiments a week or two ago, so I'm definitely a novice.

I have a few things I can play with to get the number of runs down a decent amount (for example, some variables are only applicable when other variables have certain values) but without some sort of fractional factorial algorithm, I don't think I can get this below 100,000 tests.

My internet searches have yielded plenty of results on 2-level fractional factorial design and I think I mostly get the hang of how that works. But the few pages on multi-level factorial design go way over my head.

Could someone try to explain to me how to create a multi-level factorial design with a somewhat large number of inputs? (Note that I don't care if its a tedious process, because I'm going to write code to do all the busy work for me, I just need to understand the algorithm to create the tests).

Improve this question
$\endgroup$
2
  • $\begingroup$ The 3-level variables are categorical? Can you do with two-way interactions? Is it meaningful to fit a linear model, or do you only want to check against that program defining "truth"? $\endgroup$
    – kjetil b halvorsen ♦
    Commented Mar 20, 2017 at 23:11
  • $\begingroup$ @kjetil, good questions. The three level variables are categorical. Since the object is to test for bugs in the program, I have to assume that unknown bugs may be present and therefore there's no telling how involved the interactions may be. But I think that limiting it to two-way interactions would still probably be ok. I am only trying to compare the two models to see if they are close enough, and am not trying maximize anything, so I don't think I need to worry about trying to fit a linear model. $\endgroup$
    – NateW
    Commented Mar 21, 2017 at 14:26

1 Answer 1

Reset to default
5
$\begingroup$

Since the purpose of the design is to "cover" the input space of the program to search for bugs, not to optimize or fit some model, it is not clear that statistical design of experiments is useful. See https://cran.r-project.org/web/views/ExperimentalDesign.html for an overview of what is available in R, especially the section "Experimental designs for computer experiments ". Latin hypercube sampling from the package lhs is sometimes used for computer experiments, but it is designed for continuous variables, not for factors. Since you have factors with two or three levels, you could maybe use n=6:

library(lhs)
maximinLHS(n=6,k=10)
          [,1]       [,2]       [,3]       [,4]       [,5]       [,6]
[1,] 0.1581259 0.34922240 0.77650438 0.57445546 0.39810199 0.53289942
[2,] 0.3942601 0.71557361 0.48436302 0.01056952 0.52782167 0.75466255
[3,] 0.5420127 0.08628741 0.59935565 0.45518451 0.31659314 0.41713492
[4,] 0.9704732 0.22435911 0.09793048 0.26863924 0.69497397 0.23899666
[5,] 0.7315111 0.97137094 0.85733108 0.72832007 0.06766873 0.04235271
[6,] 0.2159874 0.50054039 0.19493044 0.96520339 0.89314903 0.90171824
           [,7]       [,8]      [,9]      [,10]
[1,] 0.94280352 0.64377198 0.5827483 0.92633749
[2,] 0.43391370 0.09490557 0.6728905 0.06741415
[3,] 0.21854447 0.78748899 0.1438589 0.30366601
[4,] 0.70834668 0.29569709 0.3194497 0.68048008
[5,] 0.52475737 0.46038173 0.8618724 0.48145297
[6,] 0.08212028 0.95824574 0.4378693 0.52659065

and then divide the unit interval in two halves for the two-level factors, in three for the tree-level factors. Maybe. There are other ideas in that section of the task view.

Returning to the question for fractional factorial designs, multiple R packages can be used. But most will only generate symmetric designs where all factors have the same number of levels. One exception is the package planor, its use seems complicated. Below I give an example of use of AlgDesign, which is simpler to use:

library(AlgDesign)
cand  <-  gen.factorial(levels=c(rep(2,3),rep(3,3)),
                        nVars= 6,
                        factors="all", varNames = LETTERS[1:6])
des  <-  optFederov( ~ ., data=cand, nTrials = 36)
ev   <-  eval.design( ~ ., des$design, confounding=TRUE, X=cand)
Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Thank you. I'm accepting this answer mostly because you pointed out that I'm really just trying to cover the input domain and not necessarily trying to create an optimizing function. I decided against lhs because in this case it would only consist of 6 runs and I think I would benefit more from a few thousand. However, the link got me thinking and exploring and it looks like there are lots of good options out there. In the end, I'm going with JMP's "space filling design". $\endgroup$
    – NateW
    Commented Mar 22, 2017 at 15:46
  • $\begingroup$ Do you know if there is another package (available) for multi level designs? What about CCD, isn't it a kind of multi level? $\endgroup$
    – Ben
    Commented Jul 22, 2022 at 12:55
  • $\begingroup$ There is a function to generate ccd in the R package rsm on CRAN. $\endgroup$
    – kjetil b halvorsen ♦
    Commented Aug 20, 2022 at 18:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.