That wholly depends on the model's purpose and your notion of accuracy:
Small/zero residuals are fine, if you are just interested in fitting a line through the test data points (e.g. using splines).
They are bad, if you want to make predictions based on the model in a random test setting, as they suggest heavy overfitting. In this case, you are much better off with looking at the "residuals vs. fits" plot, in order to look for potential problems with the model (outliers, patterns, non-symmetry around zero, etc.).
EDIT: As you used a (different) training set for fitting the model, things are a little bit different:
Overfitting is obviously not an issue here, assuming the sets were
independently drawn. But still, the absolute size of the residuals
by itself doesn't say much, as was pointed out by chRrr - if the
variance is very small, the residuals will naturally be smaller, too.
Hence, they are affected by scaling.
When comparing different models, fitted on the same training sample,
a smaller $RSS$ (meaning "smaller residuals") on the test sample
certainly is preferable.
If you are fitting a linear model, you can also go with the $R^2$, in order to link the size of the residuals ($RSS$) with the test data variation.
In summary: The absolute size of test data residuals alone doesn't say anything about model accuracy. However, it can be used to assess the quality of a model, when comparing it relative to other values, like the test data variation (for linear models) or other models' residual size.