If you have a fitted GARCH model, you want the stardardized innovations to be i.i.d. Therefore, you take their sample counterparts, standardized residuals, and conduct the Li-Mak test. The test inspects presence of autocorrelation in their squares. Presence of autocorrelation (high test statistic, low $p$-value, and rejection of the null hypothesis of no autocorrelation in squares) is a sign that your GARCH model has failed to capture all autoregressive conditional heteroskedastic patterns there are.

Looking at the description of the Weighted.LM.test function in R, I see it takes residuals x and conditional variances h.t as inputs. I presume it wants x to be the raw residuals that can be scaled by dividing by the square root conditional variances sqrt(h.t) so that standardized residuals are obtained.

Brief answer to the question: raw, nonstandardized residuals should be used.

If you have a fitted GARCH model, you want the stardardized innovations to be i.i.d. Therefore, you take their sample counterparts, standardized residuals, and conduct the Li-Mak test. The test inspects presence of autocorrelation in their squares. Presence of autocorrelation (high test statistic, low $p$-value, and rejection of the null hypothesis of no autocorrelation in squares) is a sign that your GARCH model has failed to capture all autoregressive conditional heteroskedastic patterns there are.

The description of the Weighted.LM.test function in R suggests that it takes residuals x and conditional variances h.t as inputs. I presume the function treats x as raw residuals that can be scaled by dividing by the square root conditional variances sqrt(h.t) so that standardized residuals are obtained. And indeed, there is a line of code that does that (or actually almost that) at the beginning of the function: x <- x^2/h.t. These x are squares of standardized residuals that are further used in calculating the autocorrelations and using the latter to construct the test statistic(s).