If you are only interested in outliers in one variable $X$, you can simply estimate the probability density $f(x)$ of this variable from the data and define outlieres as values $x$ with $$ \alpha/2 > P(X<x) = \int_{-\infty}^x f(x')\, dx'$$ For large values the citerion is anlogous, i.e.: $P(X>x)<\alpha/2$.
Estimation of $f(x)$ can be done either parametric (e.g. a normal density) or non-prametric (kernel density estimator), as provided by the R function density(). In the latter case, the integration must be done numerically, for example with Simpson's rule (this has the advantage of working with equidistant function sample points).
If the distribution is multimodal and outliers can lie in the middle between maxima, it might be the simplest to base outlier detection on distance statistics. See section 5 "Distance rejection" in this report.