I perform a permutation test multiple times on different datasets, each time I am only concerned about significant $p$ values. To reduce computation time would it be correct to introduce this kind of stopping rule: After a certain number of $N$ permutations to check whether $p$ is greater than a particular value. So, for example if $p>0.1$ after $N$=200 permutations then the lower bound of 95% confidence interval would be greater than $0.05$. Therefore, calculations could be stopped as a true $p$ is not significant. Just want to make sure I am doing it right. Thank you.
If it's not quite significant, do you keep simulating?
If so, then no, the significance level is no longer the desired level when you do that. It's substantially affected by this checking for significance along the way.
You can adjust things so that the properties are what you want, but it's not a matter of just stopping early.
One way of doing this is called sequential probability ratio testing (SPRT).