Brief answer to the question: raw, nonstandardized residuals should be used.
If you have a fitted GARCH model, you want the stardardized innovations to be i.i.d. Therefore, you take their sample counterparts, standardized residuals, and conduct the Li-Mak test. The test inspects presence of autocorrelation in their squares. Presence of autocorrelation (high test statistic, low $p$-value, and rejection of the null hypothesis of no autocorrelation in squares) is a sign that your GARCH model has failed to capture all autoregressive conditional heteroskedastic patterns there are.
The description of the Weighted.LM.test
function in R suggests that it takes residuals x
and conditional variances h.t
as inputs. I presume the function treats x
as raw residuals that can be scaled by dividing by the square root conditional variances sqrt(h.t)
so that standardized residuals are obtained. And indeed, there is a line of code that does that (or actually almost that) at the beginning of the function: x <- x^2/h.t
. These x
are squares of standardized residuals that are further used in calculating the autocorrelations and using the latter to construct the test statistic(s).