# What is a good way of estimating the dependence of an output variable on the input parameters?

I am evaluating a scenario's output parameter's dependence on three parameters: A, B and C. For this, I am conducting the following experiments:

• Fix A+B, Vary C - Total four sets of (A+B) each having 4 variations of C
• Fix B+C, Vary A - Total four sets of (B+C) each having 3 variations of C
• Fix C+A, Vary B - Total four sets of (C+A) each having 6 variations of C

The output of any simulation is the value of a variable over time. For instance, A could be the area, B could be the velocity and C could be the number of vehicles. The output variable I am observing is the number of car crashes over time.

I am trying to determine which parameter(s) dominate the outcome of the experiment. By dominate, I mean that sometimes, the outcomes just does not change when one of the parameters change but when some other parameter is changed even by a small amount, a large change in the output is observed. I need to capture this effect and output some analysis from which I can understand the dependence of the output on the input parameters. A friend suggested Sensitivity Analysis but am not sure if there are simpler ways of doing it. Can someone please help me with a good (possibly easy because I don't have a Stats background) technique? It would be great if all this can be done in R.

Update: I used linear regression to obtain the following:

lm(formula = T ~ A + S + V)

Residuals:
Min       1Q   Median       3Q      Max
-0.35928 -0.06842 -0.00698  0.05591  0.42844

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.01606    0.16437  -0.098 0.923391
A            0.80199    0.15792   5.078 0.000112 ***
S           -0.27440    0.13160  -2.085 0.053441 .
V           -0.31898    0.14889  -2.142 0.047892 *
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.1665 on 16 degrees of freedom
Multiple R-squared: 0.6563, Adjusted R-squared: 0.5919
F-statistic: 10.18 on 3 and 16 DF,  p-value: 0.0005416


Does this mean that the output depends mostly on A and less on V?

-
can you really fix velocity and number of vehicles and vary the area? It does not look like it is possible to do controlled experiment in your case, can you please clarify? –  mpiktas Mar 8 '11 at 8:47
@mpiktas It seems it is a computer simulation of traffic, so anything is possible. –  mbq Mar 8 '11 at 12:00
@mpiktas: Actually @mbq is right. I am doing some computer simulations of the scenarios. –  Legend Mar 8 '11 at 17:20

EDIT: After some reflection, I modified my answer substantially.

The best thing to do would be to try to find a reasonable model for your data (for example, by using multiple linear regression). If you cannot get enough data to do this, I would try the following "non-parametric" approach. Suppose that in your data set, the covariate $A$ takes on the values $A=a_1, ..., a_{n_A}$, and likewise for $B$, $C$, etc. Then what you can do is perform a linear regression on your dependent variables against the indicator variables $I(A= a_1), I(A=a_2), ..., I(A = a_{n_A}), I(B = b_1),...$ etc. If you have enough data you can also include interaction terms such as $I(A=a_1, B=b_1)$. Then you can use model selection techniques to eliminate the covariates that have the least effect.

-
@charles.y.zheng: +1 Thank you. I used linear regression and am now trying to interpret them. I updated my question with the results just in case you can take a look at it. –  Legend Mar 9 '11 at 1:42
Before doing regression on the numerical values of the variates you will want to normalize them (scale them so that they lie between 0 and 1). Otherwise the choice of units you use will affect the answer! –  charles.y.zheng Mar 9 '11 at 1:47
@charles.y.zheng: Oh! I will do that and get back. –  Legend Mar 9 '11 at 1:52
We can conclude that A is definitely playing a role. But some other variable, like B, might turn out to have a bigger effect when we consider non-linear effects (that is, when we include additional variates A^2, B^2, etc in the data). The problem is that the more variates you want to fit, the more data your need for your regression to be meaningful. –  charles.y.zheng Mar 9 '11 at 2:09
The model will be nonlinear in $A$, $B$, etc, but the regression will still be linear in the variates $A$, $A^2$, etc. What you will want to do in R is to define a new variable "A2 = A^2", then use the command "lm(formula = T ~ A + S + V + A2)" (for example). –  charles.y.zheng Mar 9 '11 at 2:44

1. Why did you go with your particular experimental design set-up? For example, fix A+B and vary C. What would you fix A + B at? If you are interesting in determining the effect of A and B, it seems a bit strange that you can fix them at "optimal values". There are standard statistical techniques for sampling from multi-dimension space. For example, latin hypercubes.

2. Once you have your data, why not start with something simple, say multiple linear regression. You have 3 inputs A, B, C and one response variable. I suspect from your description, you may have to include interaction terms for the covariates.

Update

2. Just because one of your covariates has a smaller p-value, it doesn't mean that it has the strongest effect. For that, look at the estimates of the $\beta_i$ terms: 0.8, -0.23, -0.31.
So a one unit change in $A$ results in $T$ increasing by 0.8, whereas a one unit change in $S$ results in $T$ decreasing by 0.23. However, are the units of the covariates comparable? For example, is it may be physically impossible for $A$ to change by 1 unit. Only you can make that decision.