What makes "mild deviations" mild depends on you and what you're doing. Deviations from normality (or indeed any other assumptions) will impact the significance level and the power of a one-sample t-test.
For example, a heavy tailed distribution will tend to have a true type I error rate that's lower than the one that you'd get if the data were normal with the same critical region, and somewhat lower power (higher type II error rate) partly because of that shift in type I error.
In some cases this kind of thing mightn't matter very much (I usually don't really care all that much if my type I error rate is 3% instead of 5%, as long as I have some sense of the size and direction of the impact). In other cases it might be more important.
It's possible to use simulation to make some assessment of how badly those things might be affected for various kinds and sizes of non-normality, so you can decide then whether you're prepared to deal with that.
In large samples, the normality assumption tends to become less critical (though the other assumptions - like independence - tend to remain relatively important).
If you know a good approximation to the distributional shape, sometimes another parametric assumption can be made (for example, if the distribution for this kind of data tends to be somewhat right skew, one might be able to approximate it reasonably well with a gamma distribution and use a GLM to do the test).
Some people use transformations, but there are two problems: (i) with small samples, you can't reasonably assess whether the transformation does sufficiently improve the distribution of the test statistic; and (ii) you're no longer testing for a shift in mean of the original distribution (if you're not focused on means, that may not matter, and with some additional assumptions, you may still be able to make a suitable conclusion about means). Similar comments apply to rank-based tests; if you don't care explicitly about mean-shift alternatives or you're prepared to make some additional assumptions, the fact that they're not directly testing for mean-shifts may not matter to you.
One solution if you really don't have a pretty solid idea of what the shape might be but really do want to be testing for a shift in mean is to look at a permutation test.
