If you are only interested in outliers in *one* variable $X$, you can simply estimate the probability density $f(x)$ of this variable from the data and define outlieres as values $x$ with
$$ \alpha/2 > P(X<x) = \int_{-\infty}^x f(x')\, dx'$$
For large values the citerion is anlogous, i.e.: $P(X>x)<\alpha/2$.

Estimation of $f(x)$ can be done either parametric (e.g. a normal density) or non-prametric (kernel density estimator), as provided by the R function *density()*. In the latter case, the integration must be done numerically, for example with Simpson's rule (this has the advantage of working with equidistant function sample points).