7
$\begingroup$

I'm currently struggling with how to assess the type I error of a permutation test for significance of variance term in R. The idea that I want to follow is outlined below:

Suppose we simulate data such that $$Y_{ij}=1+b_i+\epsilon_{ij}$$ where $b_i\sim \mathcal{N}(0, \sigma_b^2)$ and $\epsilon_{ij} \sim \mathcal{N}(0, \sigma_e^2)$. We can then calculate the LRT statistic for this model (testing against the alternative of $\sigma_e^2 \neq 0$) and permute the clusters (that is, the values of the $b_i$'s) to see how does the LRT vary across the different permutations.

I was, in particular, interested in the type I error of this test, i.e. what happens if we indeed have $\sigma_b^2=0$, but the issue I'm facing is that when using the lmer() function in R I get an error as I am simulating $b_i=0$ for all $i$ and I'm assuming the model becomes unidentifiable. Is there a way to make this work, as in, how should the code of the model look like?

I should point out that I'd prefer to have just hints, not full answers as this is related to something important that will be marked and I want to be a decent human being and not rely on good people on the internet.

Thanks in advance!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
6
$\begingroup$

My first hint is to use lmer from the lme4 package, not lme. When the variance components approach zero it will be more likely to converge without errors, although it may (and should) give a singular fit warning.

My second hint is to simulate $b_i$ with $\sigma_b^2$ not as identically zero, but just very small.

Improve this answer
$\endgroup$
2
  • $\begingroup$ First of all, thank You for taking the time to comment! Secondly, I had made a mistake - I did in fact used lmer(). Also, I tried to use small value such as 0.0001 for $\sigma_b^2$ and although it still gives a warning, the procedure can be carried out. Thanks once again! $\endgroup$
    – asdf
    Commented Aug 12, 2020 at 18:14
  • $\begingroup$ Happy to help. I would also play around with different sample sizes (numbers of clusters and numbers of observations per cluster) $\endgroup$
    – Robert Long
    Commented Aug 12, 2020 at 18:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.