0
$\begingroup$

So, I have two different kernel density estimators, which yield different likelihoods for a given set of data.

To compare the models, I've then simulated the input data 200 times, 1000 samples per time, every time plugging in the values in my models, so that logH = np.sum(np.log(mle)), and then calculated $\Lambda = 2\times(logL_{H_1}-logL_{H_0})$.

Now doing this over and over, gives me something that seems to approach gaussian. It's not entirely clear where to go from here though, if I want to accept/reject my alternative hypothesis at e.g. $\alpha = 0.05$. What am I missing?

enter image description here

$\endgroup$
1
$\begingroup$

It looks like you have already obtained an appropriate test statistic for your test, so all you have to do is use this to estimate the p-value of your data. The process is as follows:

  1. Simulate data-sets $\boldsymbol{x}^{(1)}, \boldsymbol{x}^{(2)}, ..., \boldsymbol{x}^{(M)} \sim \text{IID }F_{H_0}$ from the null distribution. (It says in your post that you have simulated the input data $M=200$ times, but it doesn't specify which distribution you used. Did you simulate this from the null distribution?)

  2. Calculate the test statistics $\Delta_1, \Delta_2, ..., \Delta_M$ by taking $\Delta_i \equiv 2 \times (\ln L_1 (\boldsymbol{x}) - \ln L_0 (\boldsymbol{x}))$. For large $M$, this gives you an approximation to the null distribution of the test-statistic.

  3. Estimate the p-value for the test empirically from your generated test statistics. Letting $\Delta_*$ be your observed test statistic (the one from your actual data), you have:

    $$\text{Estimated p-value} \equiv \hat{p}(\Delta_*) \equiv \frac{1}{M} \sum_{i=1}^M \mathbb{I}(\Delta_i \geqslant \Delta_*).$$

    As $M \rightarrow \infty$ this estimate should approach the true p-value of your test. So use whatever computational power you have to generate this for a large $M$. Voila!


I notice in your post that the simulated values of you test statistic are negative, which means that the null model has a higher likelihood than the alternative. If this is the case, the estimated p-value is one. Are you sure this is the right way around?

$\endgroup$
  • $\begingroup$ The question I have at hand is phrased as "Produce Monte Carlo samples from the null KDE over [a given range]. Calculate the likelihood ratio for the Monte Carlo samples, where $H_0$ uses the null KDE and $H_1$ uses the alternative KDE." I don't see how the likelihood ratio could lead to anything other than rejection of $H_1$, because obviously the model generating the data must be the best model to fit the data. Or did I miss something here? Is this in fact meant to be a panic-inducing trick question? $\endgroup$ – komodovaran_ Mar 16 '18 at 23:16
  • $\begingroup$ The actual data is not generated; it is presumably part of what you had to start with the get the KDE you are testing. In step 1 of the above algorithm, the "data-sets" being generated are values from the null distribution, which are used to find the null distribution of the test statistic. The latter is required because classical hypothesis testing calculates a p-value predicated on the null hypothesis being true. $\endgroup$ – Reinstate Monica Mar 18 '18 at 3:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.