Significance of different mean values for two processes

Question

I have an experiment that produces one measurement every time it is run. I can change something in the experimental setup, and I want to test if this change in the setup results in a change of the measured value. I am repeating the measurement multiple times for each setup, but only one measurement with one setup can be done at a time. I assume that the change in the setup is not changing the standard deviation of the measurements (but I do not know what the deviation is), and that the measurements in both setups are normally distributed around a true value.

My hypothesis is that the true values for the two processes will be different, and I would like to test this hypothesis. However, having gathered the data and found the means to be different, I am unsure how much confidence I should assign to the result. After all, the mean values could be different by chance and the true values actually identical.

What is the best way to evaluate this?

Calculating the standard deviation of the mean seems like a good start. The further the two means +/- their standard deviations are apart, the more confident I can be. But I would like to make this a bit more quantitative.

My idea is that I need to calculate the probability of getting two different mean values if I perform two times N measurements. The lower the probality, the more confidence I can have in my result. But how do I do this calculation? To me this looks like a textbook problem where I just have to know the right formula, but apparently I haven't taken the corresponding course.

(Note: no quantum physics here, everything is classical)

Answer 1

Without more details about your processes I don't know for sure, but you probably want to compute the distribution for the sum of the difference in sampled values.

If $X^1(t) \sim N(\mu_1t,\sigma^2t)$ and $X^2(t) \sim N(\mu_2t,\sigma^2t)$ then the statistic would be

$w = \frac{1}{n} \sum_{i=1}^n (X^1(t_i) - X^2(t_i))$

Which under the null hypothesis ($\mu_1 = \mu_2$) would have mean 0 and variance $v = \frac{2\sigma^2}{n}\sum_i \Delta t_i$. This is assuming that $X^1$ and $X^2$ are not correlated. If you are estimating $\sigma$ from the data you compare the standardized statistic $z = \frac{w - 0}{\sqrt{v}}$ to a $t$-distribution with $n-1$ df.

EDIT: (response to comment) If you are sampling at different intervals and processes are independent then the following SDE approach might be reasonable,

$dX_t^i = \mu_idt + \sigma dW_t, \quad i=1,2$

with $W_t$ standard Brownian Motion. This model assumes a constant drift in each process and same level of volatility. The MLE estimate for $\mu_i$ would solved by considering

$ \begin{equation} \log(\mathbf{P}(X^i_{t_{i_1}},X^i_{t_{i_2}}\ldots,X^i_{t_{i_n}})) = \sum_{k=1}^n \log(\mathbf{P}(X^i_{t_{i_{k+1}}}|X^i_{t_{i_k}})) \propto -\sum_k (\Delta X^i_{t_{i_{k+1}}} - \mu_i \Delta t_{i_{k+1}})^2 \end{equation} $

so optimizing for $\mu_i$ yields $\hat{\mu}_i = \frac{\sum_k \Delta X^i_{t_{i_{k+1}}}}{\sum_k \Delta t_{i_{k+1}}}$, where $\Delta$ is the adjacent difference operator.

The variance of $\hat{\mu}_i$ is just $v_i = \frac{\sigma^2}{\sum_k \Delta t_{i_{k+1}}}$, so you can just do a two-sample $t$-test with statistic $(\hat{\mu}_1 - \hat{\mu}_2) / \sqrt{v_1 + v_2}$.

Answer 2

I think if means of two process are $\mu_1$ and $\mu_2$ then you can test the null hypothesis $H_0:\mu_1-\mu_2=0$ against alternative hypothesis $H_1=\mu_1-\mu_2\neq0$ and if your sample reject the null hypothesis then your statement will be true.This hypotesis can be tested but you need more information to test this.First you need to know if this two processes are correlated or not.Next though you say that the standard deviation is the same for both processes you also need whether standard deviation is known to you or not.If that is not known you need to estimate that.