Significance of difference based on confidence intervals

Let's say I have 6 random variables whose distribution is parametrized by $\theta_1,\ldots ,\theta_6$ and I constructed confidence intervals for all 6 parameters. Minimal values in the confidence interval ranges for first 3 parameters are higher than maximal values for next 3 parameters. Does it allow me to say that first 3 variables are significantly higher than latter 3 without performing any test for means?

I think your question demonstrates a typical wrong way of conducting statistical analysis. The term data snooping is often used for this kind of practice.

In the traditional statistical testing framework, your hypothesis needs to be defined BEFORE data collection. The reason is that every sampled dataset is never a perfect representation of the overall population. The sampled data will have patterns due entirely by chance. If you construct and test your hypothesis based on the same data, you will always get a significant pvalue and the inference is invalid.

For example, assume you sample 4 times from a normal distribution $N(0,1)$ and see $x_1=-1,x_2=2,x_3=3,x_4=-0.5$. By looking at the data, you construct an alternative hypothesis that the 1st and 4th samples are systematically smaller than the 2nd and 3rd samples. You then test this on the same data set and you are likely to get a significant pvalue (I didn't try to calculate a pvalue but that is the point). Do you think if your conclusion will be valid?

What you need to do is to come up with a hypothesis that represent your prior belief about the 6 parameters and test it on your data. This prior belief often comes from a domain specific expert e.g medical doctor for clinical trial and banker for financial modelling etc.

Peter

• +1 for noticing that data snooping might be an issue here. Also, careful prior reasoning about the right hypothesis to test removes much of the multiplicity concerns I adressed. – Horst Grünbusch Oct 28 '14 at 17:55

Rather not.

As you might know, if the $1-\alpha$-confidence intervals don't overlap, you can conclude that the difference of the respective parameter is significant at significance level $\alpha$. So you could infer the same with your 6 parameters.

However, as you are in a situation equivalent to performing multiple tests (15, I believe), you would have to lower the single test significance level, hence, make your confidence intervals larger. If you would use Bonferroni correction, you would have to use $1-\frac{\alpha}{15}$-confidence intervals to keep the probability of at least one false significance below $\alpha$.

After this correction, your confidence intervals might overlap. If not, then the answer is yes.