I have done a lot of research on outliers, particularly when I worked on energy data validation at Oak Ridge from 1978 to 1980. There are formal tests for univariate outliers for normal data (e.g. Grubbs' test and Dixon's ratio test). There are tests for multivariate outliers and time series. The book by Barnett and Lewis on "Outliers in Statistical Data" is the bible on outliers and covers just about everything.
When I was at Oak Ridge working on data validation we had large multivariate data sets. For univariate outliers there is a direction for extremes (highly above the mean and highly below the mean). But for multivariate outliers there are many directions to look for outliers. Our philosophy was to consider what the intended use of the data is. If you are trying to estimate certain parameters such as a bivariate correlation or a regression coefficient then you want to look in the direction that provides the greatest effect on the parameter of interest. At that time I had read Mallows' unpublished paper on influence functions. The use of influence functions to detect outliers is covered in Gnanadesikan's multivariate analysis book. Of course you can find it in Barnett and Lewis also.
The influence function for a parameter is defined at points in the multivariate space of the observations and essentially measures the difference between the parameter estimate when the data point is included compared with when it is left out. You can do such estimates with each sample point but usually you can derive a nice functional form for the influence function that gives insight and faster computation.
For example in my paper in the American Journal of Mathematical and Management Science in 1982 "The Influence Function and Its Application to Data Validation" I show the analytic formula for the influence function for bivariate correlation and that the contours of constant influence are hyperbolae. So the contours show the direction in the plane where the influence function increases the fastest.
In my paper I show how we applied the influence function for bivariate correlation with the FPC Form 4 data on generation and consumption of energy. There is a clear high positive correlation between the two and we found a few outliers that were highly influential on the estimate of correlation. Further investigation showed that at least one of the points was in error and we were able to correct it.
But an important point that I always mention when discussing outliers is that automatic rejection is wrong. The outlier is not always an error and sometimes it provides important information about the data. Valid data should not be removed just because it doesn't conform with our theory of reality. Whether or not it is difficult to do, the reason why the outlier occurred should always be investigated.
I should mention that this is not the first time multivariate outliers have been discussed in this site. A search for outliers would probably lead to several questions where multivariate outliers have been discussed . I know that I have referenced my paper and these books before and given links to them.
Also when outlier rejection is discussed many of us on this site have recommended against it especially if it is done based solely on a statistical test. Peter Huber often mentions robust estimation as an alternative to outlier rejection. The idea is that robust procedures will downweight the outliers reducing their effect on estimation without the heavy-handed step of rejecting them and using a non-robust estimator.
The influence function was actually originally developed by Frank Hampel in his PhD dissertation in the early 1970s (1974 I think). His idea was actually to use influence functions to identify estimators that were not robust against outliers and to help develop robust estimators.
Here is a link to a previous discussion on this topic where I mentioned some work of mine on detecting outliers in time series using influence functions.