Assessing significance when measurements are correlated

Question

I am trying to estimate a function $f(x)$ at $x=0.1, 0.2, 0.3, 0.4, 0.5$. I make this estimate through some complex procedure based on a set of independent but noisy data (with known uncertainties). The outcome is a set of correlated estimates.

The null hypothesis of my study is that $f(x)=0$. I want to assess, based on any/all of these measurements, whether $f(x)\neq 0$ at some/any $x$.

I have run $N$ Monte Carlo simulations of the measurement procedure by repeating the measurement with random realizations of the data (since the measurement uncertainty is unknown). Now I have $N$ sets of estimates at these five points. I can use the $N$ trials to estimate the mean and variance of each $f(x)$ as well as the covariance between the different measurements.

Taking the mean and standard deviation across all of the $N$ trials, I find that each of the measurements are roughly $f(x)=-0.2 \pm 0.1$. At normal thresholds of significance, this would be a $2\sigma$ result and therefore not significantly different from 0. However, since all of the measurements are below zero, maybe the result is significant. However, I do not know how to assess this, given that all of the measurements are correlated.

Answer 1

If I understand you correctly, you want to test the hypothesis that some unknown fixed 5-dimensional vector $f$ equals $0$. You are able to sample "estimates" $y$ of this vector.

I don't know what are the properties of these "estimates". For example, if they are all biased downwards, then all samples $y$ might be negative, while the true $f$ is positive. If it is the case, then your problem has no solution.

However, it has a simple solution, if we make simplifying assumptions. Let's assume that:

• all $y$ are distributed identically and independently of each other given $f$
• expected value of all $y$ is $f$ and they have some unknown covariance matrix $\Sigma$

In this case, you can apply multivariate CLT and infer that sample mean vector $\bar{y}$ is asymptotically jointly normally distributed with mean $f$ and covariance matrix $\frac{1}{\sqrt{N}}\Sigma$.

The covariance matrix $\hat{\Sigma}$ and mean value $\hat{f}$ may be estimated as sample covariance and mean of $y$. Now you can test the hypothesis that $f=0$ with Hotelling $T^2$ statistic: compare the value $(\hat{f}-\mu)^T\hat{\Sigma}^{-1}(\hat{f}-\mu)\frac{n-p}{p(n-1)}$ with critical value of $F_{p, n-p}$, where $n$ is number of samples, $p$ is number of parameters (5), and $\mu$ is the vector of your hypothetical expected value - that is, 0.

Answer 2

I don't think that you need to estimate the distribution of $f(x)$. If you can use large $N$ values, for me, showing that less than 5% of data points are larger than $0$ is already a test within a Monte Carlo approach (not 100% sure, but almost).

You would need to use an approximation by a normal distribution if your goal was to demonstrate that the mean value of your $f(x)$ values differs from zero (here you can apply the central limit theorem to say that the mean is normally distributed). I guess that you will then be able to show that the mean differs from $0$ (as the sd of the mean will be $\frac{0.1}{\sqrt(N)})$