I am trying to estimate a function $f(x)$ at $x=0.1, 0.2, 0.3, 0.4, 0.5$. I make this estimate through some complex procedure based on a set of independent but noisy data (with known uncertainties). The outcome is a set of correlated estimates.

The null hypothesis of my study is that $f(x)=0$. I want to assess, based on any/all of these measurements, whether $f(x)\neq 0$ at some/any $x$.

I have run $N$ Monte Carlo simulations of the measurement procedure by repeating the measurement with random realizations of the data (since the measurement uncertainty is unknown). Now I have $N$ sets of estimates at these five points. I can use the $N$ trials to estimate the mean and variance of each $f(x)$ as well as the covariance between the different measurements.

Taking the mean and standard deviation across all of the $N$ trials, I find that each of the measurements are roughly $f(x)=-0.2 \pm 0.1$. At normal thresholds of significance, this would be a $2\sigma$ result and therefore not significantly different from 0. However, since all of the measurements are below zero, maybe the result is significant. However, I do not know how to assess this, given that all of the measurements are correlated.

  • $\begingroup$ You have training data consisting of noisy observations $y$ at locations $x$. You have a black box operation that takes your training data as inputs and returns an estimate $\hat{y}$ at locations $x$. You say $\hat{y}$ is "a set of correlated outputs". What is $\hat{y}$ correlated with? It is my understanding that $\hat{y}$ is a vector. $\endgroup$ – Nat May 17 '18 at 17:32
  • $\begingroup$ Following up on my comment to glagla's answer (sorry, didn't realise I'd put it there rather than the question), is it expected that variance tends the same side of 0,or would we be looking at covariance structures that have means close to 0 but mean magnitudes much larger? $\endgroup$ – ReneBt May 20 '18 at 13:38

If I understand you correctly, you want to test the hypothesis that some unknown fixed 5-dimensional vector $f$ equals $0$. You are able to sample "estimates" $y$ of this vector.

I don't know what are the properties of these "estimates". For example, if they are all biased downwards, then all samples $y$ might be negative, while the true $f$ is positive. If it is the case, then your problem has no solution.

However, it has a simple solution, if we make simplifying assumptions. Let's assume that:

  • all $y$ are distributed identically and independently of each other given $f$
  • expected value of all $y$ is $f$ and they have some unknown covariance matrix $\Sigma$

In this case, you can apply multivariate CLT and infer that sample mean vector $\bar{y}$ is asymptotically jointly normally distributed with mean $f$ and covariance matrix $\frac{1}{\sqrt{N}}\Sigma$.

The covariance matrix $\hat{\Sigma}$ and mean value $\hat{f}$ may be estimated as sample covariance and mean of $y$. Now you can test the hypothesis that $f=0$ with Hotelling $T^2$ statistic: compare the value $(\hat{f}-\mu)^T\hat{\Sigma}^{-1}(\hat{f}-\mu)\frac{n-p}{p(n-1)}$ with critical value of $F_{p, n-p}$, where $n$ is number of samples, $p$ is number of parameters (5), and $\mu$ is the vector of your hypothetical expected value - that is, 0.


I don't think that you need to estimate the distribution of $f(x)$. If you can use large $N$ values, for me, showing that less than 5% of data points are larger than $0$ is already a test within a Monte Carlo approach (not 100% sure, but almost).

You would need to use an approximation by a normal distribution if your goal was to demonstrate that the mean value of your $f(x)$ values differs from zero (here you can apply the central limit theorem to say that the mean is normally distributed). I guess that you will then be able to show that the mean differs from $0$ (as the sd of the mean will be $\frac{0.1}{\sqrt(N)})$

  • $\begingroup$ sorry, I think I have misled you with my statement "all of the measurements are below zero". I meant all of the means are below zero. $\endgroup$ – rhombidodecahedron May 14 '18 at 12:37
  • $\begingroup$ ok, one last point, why do you say that your measurements are correlated if accordingly to your results f(x) is independent of x? testing if 95% of your observations are below 0 is not enough? $\endgroup$ – glagla May 14 '18 at 17:44
  • $\begingroup$ $f(x)$ is not independent of $x$, the measurements of $f(x)$ at the different $x$'s are all correlated and I can measure the sample correlation $\endgroup$ – rhombidodecahedron May 15 '18 at 11:40
  • $\begingroup$ So do you mean that there is a covariance structure within y (where y=transform of X) based on the location in x? The values of y at x1 influence is values at x2, x3 etc? You want to determine if f(x) =0 is different to y? Could you provide more detail on that? I have possible solutions in mind if I have interpreted x correctly, but even then it is unclear what exactly x is which would influence the appropriateness of the options. $\endgroup$ – ReneBt May 17 '18 at 7:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.