I know that the likelihood ratio test statistic follows a chi-squared distribution. But is this only asymptotically? I.e. that I have to have a large sample size in order for the test to be reliable? In smaller sample sizes, am I better off using an exact method?
That depends on other assumptions holding or not.
If, for example, you are confident to be sampling from a normal distribution, then the $\chi^2$ distribution is exact.
If you do not wish to make such an assumption, then the $\chi^2$ distribution can only be established asymptotically.