Suppose we have an ordered sequence of independent random variables, $X_1, \ldots X_m, X_{m+1}, \ldots, X_n$, where under the null hypothesis, $X_{1..n}$ are identically distributed with $$\mathbb{E}[X_i] = 0.5, \:i \in \{1..n\},$$ whereas under the alternative hypothesis, the sequence is split into two groups of identically distributed random variables on the basis of an unknown parameter $m$, with $$\mathbb{E}[X_i] = 0.5, \:i \in \{1..m\}$$ and $$\mathbb{E}[X_j] > 0.5, \:j \in \{(m\!+\!1)..n\}.$$ In observing a sample $x_1, \ldots x_m, x_{m+1}, \ldots, x_n$ of this sequence, we do not know $m$.

How do we test for the alternative hypothesis?

Proposed approach and problem

It seems a reasonable approach might be to first find the maximum likelihood estimation of the unknown parameter $m$ and then use a standard test for comparing the two resulting groups. However, this skews the results in favour of the alternative hypothesis, as we explicitly picked $m$ such that it maximised the $p$-values of the test.

How do we control for the bias that was introduced when picking $m$? Is there a better approach to testing for the alternative hypothesis that avoids this issue?

I'm very much hoping for an analytical approach rather than an approach building on Monte Carlo methods.

  • $\begingroup$ I recognise an alternative approach of simultaneously testing at all possible values of $m$ and then using existing approaches for handling the resulting multiple comparison problem, but I fear loss of statistical power. $\endgroup$ – sircolinton Apr 22 at 3:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.