# Significance of different mean values for two processes

I have an experiment that produces one measurement every time it is run. I can change something in the experimental setup, and I want to test if this change in the setup results in a change of the measured value. I am repeating the measurement multiple times for each setup, but only one measurement with one setup can be done at a time. I assume that the change in the setup is not changing the standard deviation of the measurements (but I do not know what the deviation is), and that the measurements in both setups are normally distributed around a true value.

My hypothesis is that the true values for the two processes will be different, and I would like to test this hypothesis. However, having gathered the data and found the means to be different, I am unsure how much confidence I should assign to the result. After all, the mean values could be different by chance and the true values actually identical.

What is the best way to evaluate this?

Calculating the standard deviation of the mean seems like a good start. The further the two means +/- their standard deviations are apart, the more confident I can be. But I would like to make this a bit more quantitative.

My idea is that I need to calculate the probability of getting two different mean values if I perform two times N measurements. The lower the probality, the more confidence I can have in my result. But how do I do this calculation? To me this looks like a textbook problem where I just have to know the right formula, but apparently I haven't taken the corresponding course.

(Note: no quantum physics here, everything is classical)

-
An important phrase here "the standard deviation of the mean" which is not the same as the standard deviation of the distribution. –  dmckee Aug 23 '12 at 17:09
(1) Are the processes and their measurements physically independent or not? (2) Are the processes perhaps simultaneously measured, so that there is a one-to-one correspondence (determined by measurement time) between the sets of measurements? –  whuber Aug 23 '12 at 18:34

## migrated from physics.stackexchange.comAug 23 '12 at 17:42

This question came from our site for active researchers, academics and students of physics.

Without more details about your processes I don't know for sure, but you probably want to compute the distribution for the sum of the difference in sampled values.

If $X^1(t) \sim N(\mu_1t,\sigma^2t)$ and $X^2(t) \sim N(\mu_2t,\sigma^2t)$ then the statistic would be

$w = \frac{1}{n} \sum_{i=1}^n (X^1(t_i) - X^2(t_i))$

Which under the null hypothesis ($\mu_1 = \mu_2$) would have mean 0 and variance $v = \frac{2\sigma^2}{n}\sum_i \Delta t_i$. This is assuming that $X^1$ and $X^2$ are not correlated. If you are estimating $\sigma$ from the data you compare the standardized statistic $z = \frac{w - 0}{\sqrt{v}}$ to a $t$-distribution with $n-1$ df.

EDIT: (response to comment) If you are sampling at different intervals and processes are independent then the following SDE approach might be reasonable,

$dX_t^i = \mu_idt + \sigma dW_t, \quad i=1,2$

with $W_t$ standard Brownian Motion. This model assumes a constant drift in each process and same level of volatility. The MLE estimate for $\mu_i$ would solved by considering

$$$\log(\mathbf{P}(X^i_{t_{i_1}},X^i_{t_{i_2}}\ldots,X^i_{t_{i_n}})) = \sum_{k=1}^n \log(\mathbf{P}(X^i_{t_{i_{k+1}}}|X^i_{t_{i_k}})) \propto -\sum_k (\Delta X^i_{t_{i_{k+1}}} - \mu_i \Delta t_{i_{k+1}})^2$$$

so optimizing for $\mu_i$ yields $\hat{\mu}_i = \frac{\sum_k \Delta X^i_{t_{i_{k+1}}}}{\sum_k \Delta t_{i_{k+1}}}$, where $\Delta$ is the adjacent difference operator.

The variance of $\hat{\mu}_i$ is just $v_i = \frac{\sigma^2}{\sum_k \Delta t_{i_{k+1}}}$, so you can just do a two-sample $t$-test with statistic $(\hat{\mu}_1 - \hat{\mu}_2) / \sqrt{v_1 + v_2}$.

-
I think what needs explaining is why this is the appropriate null hypothesis to use. The appearance of the multiple $t$ in your model is a puzzle, though: why such a formulation? –  whuber Aug 23 '12 at 18:31
I'm happy to describe the process in more detail - but what details do you need? –  Nikratio Aug 23 '12 at 18:44
@whuber I agree without further information these might be poor assumptions. The description the OP gave seemed to indicate a BM with drift type model sampled at identical times. –  muratoa Aug 23 '12 at 18:45
Just extended the original question with more details. I believe (1) processes are uncorrelated, (2) not simultanously measured. –  Nikratio Aug 23 '12 at 18:51
I don't see where there's a Brownian motion here. Along with the OP, I see a textbook example calling for an unpaired t-test based on the data, not their differences. –  whuber Aug 23 '12 at 19:58
I think if means of two process are $\mu_1$ and $\mu_2$ then you can test the null hypothesis $H_0:\mu_1-\mu_2=0$ against alternative hypothesis $H_1=\mu_1-\mu_2\neq0$ and if your sample reject the null hypothesis then your statement will be true.This hypotesis can be tested but you need more information to test this.First you need to know if this two processes are correlated or not.Next though you say that the standard deviation is the same for both processes you also need whether standard deviation is known to you or not.If that is not known you need to estimate that.