I have 2 input variables, $X_1$ and $X_2$ that affect output variable $Y$. I can run experiments where I modify the inputs and measure what happens to the output.

Now, if $X_1$ and $X_2$ were binary, I could just run four sets of experiments, collect $Y$ and use t-test to figure out if any combination affects output differently. Most importantly i can use power tables to check what should the sample size for each experiment be in order to achieve a desired power level.

However, if $X_1$ and $X_2$ are continuous between 0 and 1 I am not sure how to compute sample size and power. I am willing to assume some form of continuous function connecting $Y$ to $f(X_1,X_2)$ although not necessarily a linear one, but I am unclear about what my power is when I check, for example, if the difference between $f(X_1,X_2)$ and $f(X_1,X_2+h)$ is significant or not.
And given this, can I figure out what the sample size ought to be?

  • 1
    $\begingroup$ Exactly what are you trying to accomplish? It sounds something like response surface modeling, but then power is not a principal concern. $\endgroup$ – whuber Oct 17 '19 at 11:42
  • $\begingroup$ Well at the end of the day I want to know how sure I am whether f(x) and f(x') are different (for arbitrary x, x'). I want to run just enough simulations (probably sampling at random uniformly around x, x') to achieve a given level of confidence $\endgroup$ – CarrKnight Oct 17 '19 at 13:14
  • $\begingroup$ Your question is ill-defined: what would "arbitrary x, x'" possibly mean? If randomly chosen, sometimes they will be so close that no amount of sampling could possibly distinguish the responses. $\endgroup$ – whuber Oct 17 '19 at 13:34
  • $\begingroup$ Exactly! is there a function/table connecting distance between X and sample size needed to be sure the response are different when the regression says they are? $\endgroup$ – CarrKnight Oct 17 '19 at 15:07
  • 1
    $\begingroup$ When you do the experiment with just two values of $X,$ yes there is such a function. In general there is not: it depends on all the values of $X$ you use, on the nature of $f,$ and even on the experimental design (e.g., whether it is adaptive). You can't figure out sample size until you have specified the experiment in full detail. $\endgroup$ – whuber Oct 17 '19 at 15:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.