I have an experiment with any possible (reasonable) number of parameters (independent variables). I run the experiment several times for each possible combination of my variables. The data I get will be generally numeric.

However I know nothing (and any assumptions are difficult) about the distribution of my data.

What I am interested in is a measure of how well do my parameters predict the data I get. Which statistic should I use? How do I calculate it (by hand, a link to a tutorial would be very sweet)?


I am trying to solve this as generally as possible (hence the slightly non-specific description) for a piece of software I'm working on. To make it a bit more clear a bit of an example:

I have these parameters:

decay: 0.1 | 0.2 | 0.3
particles: 10 | 100
velocity: 30 | 70

This gives 12 combinations (3 * 2 * 2) and I'll measure my dependent variable (say temperature) five times for each combinations.

Thus my final dataset will have 60 measurements of temperature.

Now suppose that temperature was in fact given by:

$t = K(0.3v + 0.6d + \varepsilon)$

where $t$ is temperature, $K$ is some constant, $v$ is velocity, $d$ is decay and $\varepsilon$ is some sort of random effect. Particles is completely unrelated to the measured temperature.

Now I'd like to perform a test that would tell me that velocity has a ~0.3 effect, decay ~0.6 and particles ~0 effect.

However I may have more or less variables and more or less measurements.

  • $\begingroup$ It sounds like it is indeed the topic of design of experiments. However, your description lacks some information; e.g., what's the sample size, are all variables crossed, is this a repeated-measures design (i.e., data are collected on the same statistical units); how many levels have your variables, and most importantly, are you interested in a predictive or an explicative model (from what you described, your use of the term 'predict' is not obvious)? $\endgroup$ – chl Dec 4 '10 at 19:58
  • $\begingroup$ @chl I'm honestly a bit lost with this, but I've updated the question with an example of what I'm trying to achieve. If that's not enough I'll try to expand it a bit more. $\endgroup$ – Jakub Hampl Dec 4 '10 at 20:15
  • $\begingroup$ So, do you already assume a linear relationship between your variables (like the function you cooked up for $t$) or is that also part of what you want to determine? $\endgroup$ – Raskolnikov Dec 4 '10 at 20:43
  • $\begingroup$ That is also part of what I want to determine. However I don't necessarily need extremely precise results - a linear approximation would probably be okay... $\endgroup$ – Jakub Hampl Dec 4 '10 at 20:48

Well, following your update, it seems you are dealing with a factorial experiment (factorial means that every factors are crossed, or, in other words, each unit is subjected to every possible combination of your factors), with five replicates. Let assume that these are not the same statistical units whose temperature is repeatedly measured across each of the 12 combinations (for the sake of clarity).

An ANalysis Of VAriance (ANOVA) seems to be the most appropriate method to deal with this design. Basically, it will allow you to estimate the contribution of each source of variance (decay, particles, and velocity) wrt. the total variance in the observed temperature. What is not explained by these factors is called the residual variance (what you call the 'random effect'). A full additive model (i.e., without modeling interaction between your factors) will read something like

$$ y_{ijkl}=\mu + \alpha_i + \beta_j + \gamma_k + \varepsilon_{ijkl}, $$

where $y_{ijkl}$ is the temperature for unit $l$ when considering levels $i=1\dots a$, $j=1\dots b$, and $k=1\dots c$, of factors $\alpha$ (decay), $\beta$ (particles), and $\gamma$ (velocity); the $\varepsilon_{ijk}$ are the residuals assumed to follow a gaussian distribution of unknown variance, $\sigma^2$. They can be viewed as random fluctuations around $\mu$, the overall mean, and reflect the between-unit variations that are not accounted for by the other factors. The $\alpha_i$, $\beta_j$, and $\gamma_k$ can be viewed as factor-specific deviations from the overall mean $\mu$.

The so-called main effect of decay, particles, and velocity will be estimated by forming a ratio between the variance that they account for (known as mean squares) and the residual variance (what is left after considering all variance explained by those factors), which is known to follow a Fisher-Snedecor (F) distribution, with $d-1$ and $N-abc$ degrees of freedom, where $d=a$, $b$, or $c$ stands for the number of levels of $\alpha$ (decay), $\beta$ (particles), and $\gamma$ (velocity). A significant effect (following an hypothesis test of a null effect, i.e. $H_0:\, \mu_i=\mu_j\,\, \forall i\neq j$ vs. $H_1:$ at least two of the $\mu_i$'s differ) would indicate that the factor under consideration has a significant effect on the outcome. This is readily obtained by any statistical software. For instance, in R you would use something like

summary(aov(temperature ~ decay + particles + velocity, data=df))

provided temperature and factor levels are organized in four columns, in a data.frame named df, as suggested below:

t1 0.1 10 30
t2 0.1 10 30
t3 0.1 10 30
t4 0.1 10 30
t5 0.1 10 30
t6 0.2 10 30
t7 0.2 10 30
t60 0.3 100 70

The effect of any of the three factors can also be summarized under an equation like the one you referred to sy simply calling (again under R):

summary.lm(aov(temperature ~ decay + particles + velocity))

This follows from the fact that an ANOVA is nothing more than a Linear Model that you may have heard about (think of a regression model where the explanatory variables are all categorical).

Should you want to account for possible interactions between all three factors, you need to add three second-order and one three-order interaction terms. If any of these effects prove to be significant, this would mean that the effect of the corresponding factors cannot be considered in isolation one from the other (e.g., the effect of decay on temperature is not the same depending on the number of particles).

As for references, I would suggest starting with on-line tutorial or textbook like Three-Way ANOVA, by Barry Cohen, or Practical Regression and Anova using R, by John MainDonald (but see also other textbooks available on CRAN documentation). The definitive reference is Montgomery, Design and Analysis of Experiments (Wiley, 2005).

  • $\begingroup$ Thanks for the detailed answer. I've heard that ANOVA's aren't practical for more then 3 independent variables. Is this true? If so, why? $\endgroup$ – Jakub Hampl Dec 4 '10 at 21:26
  • 2
    $\begingroup$ @honitom It really depends on the context of your experiment and your sample size. Keep in mind that the reliability of your estimates (the regression coefficients summarizing the effect of your factors on your outcome) depends on the available sample size and associated variance. In DoE, interaction terms can be aliased with main effects, depending on the sample size; still we can estimate the main effects. Any decent textbook should cover this topic much well than me in a comment, so I'd suggest to have a look at ANOVAs and DoE. $\endgroup$ – chl Dec 4 '10 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.