Trends in noisy data and the applicability of the 95% confidence interval

Question

I am performing simulations while measuring a quantity A which depends on the parameter B. I make N independent measurements of A for given values of B. I can then calculate the mean to get an estimation of what the real value of A is. My question is about the error.

I can calculate the 95% confidence interval as 1.96xstddev(A)/sqrt(N) for each value of B. What I often find however is that this is not really representative of the error. For example, if I have the following data:

B   A(B)  95%CI
1   2     10
2   4     10
3   6     10
4   8     10

Without knowledge of the error, one would conclude that there is a linear relationship between B and A. And in my experience, as I make more measurements, this relation doesn't change, despite the noise in the measurement. But if I show this result to someone else, they say "given those error bars (+-10 on each point), you can't say anything about the trend".

Since confidence intervals assume a normal distribution, are my measurements somehow more-sharply peaked somehow? Or am I using the 95%CI wrongly? Is there a better way to show the confidence in my data and/or the trend? Any advice is greatly appreciated.

Answer 1

Since confidence intervals assume a normal distribution, are my measurements somehow more-sharply peaked somehow?

That is a possibility. However, if they're more sharply peaked, it usually means that the tails are fatter, i.e. you would underestimating the width of confidence intervals, take a look at leptokurtic distributions.

If your error distribution is not normal then the confidence intervals would not be correct. In physical sciences the errors are usually normal for continuous variables though. You can test for normality by using Jarque-Bera or similar tests.

Or am I using the 95%CI wrongly?

You are using it correctly. However, you mentioned that you did other measurements, and that relationship holds. So, why don't you collect more data? You'll see that $\frac{1}{\sqrt{N}}$ will gradually decrease the width of your confidence bands. If you are right, then all it takes is to get the bigger sample size.

Is there a better way to show the confidence in my data and/or the trend? Any advice is greatly appreciated.

You seem to think that the relationship between A and B is linear. In this case maybe you should run a simple linear regression like $A_i=\beta_0+\beta_B B_i+\varepsilon_i$, where you lump all your measurements together, and get the slope of the regression line, in your example it would be $\beta_B=2$. The regression will also give you the confidence bands for $\beta_B$, and not only the error variance.

If you think that the error variance could be different for different B, then you could use mixed effect regression. I would start with simple linear regression and see how it works though.

Answer 2

the 95% Confidence interval (calculated by the formula you have given) is around the MEAN(A). Better way is to normalize the variable and then perform 95% CI test.

In general, the 95% CI test if you perform the experiment N times with A (and B), then 95% of the time you will find the values of A at +/-1.96xSTD(A)/SQRT(N) around MEAN(A).

If A is indeed normally distributed then the sharp peak must be at MEAN(A) or at ZERO (if you normalize A). But since you are testing a linear relationship between A and B, I would also recommend testing if B is also normally distributed.

I hope it helps you further in your experiments.