The 4 doesn't pose any problem. Just subtract 4 from every observation and you can test $\mu=0$ on the shifted data.
There are two common nonparametric tests of location which might be relevant to you -- the Wilcoxon signed rank test and the sign test.
The signed rank test is a test for the pseudo-median. When accompanied by the assumption of symmetry (which the signed rank test needs for the permutations to be equally likely under the null), the population pseudo-median will equal the population median, and (by symmetry) will also equal the population mean when it is finite (which is hereafter assumed to save repeating it every time I mention the mean). Which is to say, under the usual assumptions the signed rank test is a suitable test for a mean as well as a median (keeping in mind that strictly that assumption is only required when the null is actually true).
The sign test doesn't require the assumption of symmetry to be a valid test but it's a test for the median rather than the mean, so you'll still need an assumption that would make the two equal (for which symmetry is sufficient but not necessary, since many asymmetric distributions can also have mean=median).
There's also the possibility of performing a permutation test based on the mean. This will be similar to the signed rank test but performed on the original observations rather than the ranks. In order that the signs can be permuted under the null, you would (again) assume symmetry.
Josh suggested the bootstrap as another possibility, and that should work quite well (though it will tend to have closer to the desired significance level if you have fairly large samples). It should not require any assumption of symmetry.
Another possibility is a different parametric assumption (one more suited to the situation you're sampling, which you haven't stated). For example you might assume that the data are drawn (say) from an exponential distribution, and test whether the mean is 4 against the alternative that it is different from 4.
Finally if the distribution is not very far from normal you might simply proceed with the t-test. [While the significance level might not be badly affected by the non-normality, the power can be if you're not at least reasonably close to normality. This is not an issue that goes away with larger samples.]