# Does the presence of an outlier increase the probability that another outlier will also be present on the same observation?

**Edit: (10/26/13) More clear (hopefully) mini-rewrites added at the bottom**

I'm asking this from a theoretical/general standpoint - not one that applies to a specific use case.

Assuming the data does not contain any measurement errors, if you're looking at a specific observation in your data and one of the measurements you recorded contains what could be considered an outlier, does this increase the probability (above that of the rest of the observations that do not contained measured outliers) that the same observation will contain another outlier in another measurement?

For my answer I'm looking for some sort of theorem, principle, etc. that states what I'm trying to communicate here much more elegantly. For clearer explanations see Gino and Behacad's answers.

Example:

Let's say you're measuring the height and circumference of a certain type of plant. Each observation corresponds to 1 plant (you're only doing this once).

For height, you measure:

Obs 1  |   10 cm
Obs 2  |   9 cm
Obs 3  |   11 cm
Obs 4  |   22 cm
Obs 5  |   10 cm
Obs 6  |   9 cm
Obs 7  |   11 cm
Obs 8  |   10 cm
Obs 9  |   11 cm
Obs 10  |   9 cm
Obs 11  |   11 cm
Obs 12  |   10 cm
Obs 13  |   9 cm
Obs 14  |   10 cm


Since observation 4 contains what could be considered an outlier from the rest of the data, would the probability increase that measured circumference also contains an outlier for observation #4?

I understand my example may be too idealistic but I think it gets the point across...just change the measurements to anything.

# Edited in attempts to make more clear:

(10/26/13)

Version 1 Attempt of Abbreviated Question:

In nature and in general, is there a tendency (even a weak one) that the probability is greater that the "degree of variance from the mean in any attribute(s) of an observation" will be similar to the "degree of variance from the mean in any other* specific attribute of that same observation" in comparison to the probability that it will INSTEAD be more similar to the "degree of variance from the mean in that same* specific attribute of any other observation."

* next to a word means I was pairing what they reference.
"Quotes" used above mean nothing and are used simply to help section parts together/off for clarity.

Version 2 Attempt of Abbreviated Question:

In nature and in general, is variance from the mean across observations for one attribute¹ correlated (even with extremely loose correlation) to the variance from the mean across observations for all attributes¹?

¹Attribute meaning measurement, quality, presence-of-either of these, and/or nearly anything else that the word "attribute" could even slightly represent as a word. Include all synonyms of the word "attribute" as well.

• Unless you stipulate a definite probability model for your data and also define--in a specific and operational way--what "outlier" might mean, this question will not be answerable. – whuber Oct 23 '13 at 18:25
• If the variables are correlated in the tails, then you will have such tendency. However, since outliers are rare, there is no realistic empirical way to show such 'tail dependency'. – Michael M Oct 23 '13 at 18:43
• I hardly know where to begin. Let's start with getting some context for the question and understanding the meanings of the terms you use. Precisely what do you mean by the "probability" of an outlier? Are you supposing a sequential experiment in which the chance of an "outlier" changes over time? If so, what is your model for this? Do you perhaps mean your estimated probability of an outlier? If the latter, what was your initial estimated probability (or basis to assess a change in probability)? How are you estimating the probability? How are you identifying "outliers"? – whuber Oct 23 '13 at 19:39
• If the variables are independent, then no. If they're dependent, it depends on the form of dependence and how that interacts with your definition of what makes an outlier. – Glen_b Oct 24 '13 at 0:39
• If they're dependent, it's almost certain to change the probability, but it may be up or down and it may be a lot or a little; we can't say much of anything in the general case. – Glen_b Oct 24 '13 at 1:10

NARROW VIEW 1
A narrow view of this question (since the other answers and comments I believe have covered adequately various general approaches) is "Assume two random variables $X$ and $Y$ are dependent. Does the variance of $X$ conditional on $Y$ is a function of the variance of $Y$?

To take refuge in the normal distribution, if both $X$ and $Y$ are normal and dependent, denote $\sigma^2_x$, $\sigma^2_y$, $\sigma_{xy}$ their unconditional variances and their covariance respectively. Then

$$\operatorname {Var}(X\mid Y) = \sigma^2_x-\frac {\sigma_{xy}^2}{\sigma^2_y}$$

which is increasing in $\sigma^2_y$. Note that the direction of their covariance (positive/negative) doesn't matter.

But also, note that the conditional variance is lower than the unconditional variance...

NARROW VIEW 2
A 2nd narrow view of the matter is: consider pairs of random variables $(X_i,Y_i),\; i=1,...,n$, and $\sigma^2_x(i)$ and $\sigma^2_y(i)$ their unconditional variances. Assume that for some indices $k,j \in [1,...,n]$, we have $\sigma^2_y(k) > \sigma^2_y(j), \forall j\neq k$.
Should we "expect" that $\sigma^2_x(k) > \sigma^2_x(j), \forall j\neq k$ also?
This formalization of the OP's question requires from us to consider directly the dependence of each $(X_i,Y_i)$ on a common source, say a vector of random variables $\mathbf Z_i$, So we have

$$X_i = h_i(\mathbf Z_i),\;\; Y_i = g_i(\mathbf Z_i)$$

Then we ask "does (or when) $$\operatorname {Var}[g_k(\mathbf Z_k)]>\operatorname {Var}[g_j(\mathbf Z_j)] \Rightarrow ? \operatorname {Var}[h_k(\mathbf Z_k)]>\operatorname {Var}[h_j(\mathbf Z_j)],\; \forall j\neq k$$

To be able to tell something about such an inequality, it must be the case that at least some of the variables that appear in $Z_j$, must also appear in the other $Z_i$'s: then as a necessary condition, we do not only require that the elements of each pair of $(X_i,Y_i)$ rv's are dependent -we require also that the pairs are dependent between them: good-bye i.i.d samples...

• @Alecos_Papadopoulos Not gonna lie, this is a SICK answer - and the type I've genuinely been looking for - as I can't write a sort of proof like that. But it comes from you with unintended consequences. I've actually noticed that as I explain this to anyone I know - those who are actually less mathematically inclined surprisingly understand it and agree with it in comparison to those who do - as those who are mathematically inclined can't help but put generalized labels on things as a matter of habit (afterall, labels do make things much easier and efficient - the problem is they ...cont.. – Taal Oct 31 '13 at 9:29
• ..cont.. also introduce artificiality. Before I go deeper, I want to clarify I'm speaking of the idea of an "independent variable." No variable is truly "indepedent" in nature/the universe- however we must assume some if not most are "indepedent" because it's not feasible to incorporate all potential permutations and level of complexity of thus in a manner that is practical in reality, we must actually make this assumption. – Taal Oct 31 '13 at 9:35
• The last part "good-bye i.i.d samples..." - yes - without sarcasm this time though - truly indeed :) Goodbye i.i.d samples. You don't exist. Keep in mind though, I'd still say there's a greater tendency for my conjecture to appear or manifest itself (but not a greater one for it to be true...as it's always true as a tendency across nature/universe) the more the dependence between variables is obvious/intense. – Taal Oct 31 '13 at 9:40
• There has been a time where Science was subsumed in Philosophy. In the old days, people instinctively felt that "everything is connected to everything else", and built their attempts to understand the world around this principle, and trying to make it operational. Rationalism emerged because the human race proved slow and sloppy in this herculean task: and rationalism means narrowing your view in order to obtain applicable results. It is an ugly compromise, I know, but it was the price to pay in order for the human race to rule the world -a world it fundamentally misunderstands. – Alecos Papadopoulos Oct 31 '13 at 10:14
• Exactly. Although you don't fully admit above to the first concept that "everything is connected to everything else," you do at least insinuate it :) I only say this because you agree with me, as I've said in your comments and a few others I just made that we put these labels for the sake of practicality. Thus, simply by process of elimination I'm presuming you do agree - but I'd be curious as to your true opinion, because if so...it looks like we have a proof pending one assumes there is some connection between everything. To me, I'm not sure how that's not true - but can't prove it. – Taal Oct 31 '13 at 10:59

It might be true for your example (or it might not—plant No. 4 could be an etiolated seedling elongating its stem in a hunt for light without expanding its girth), but it's a matter of prior botanical knowledge rather than any general statistical law. It's not hard to construct counter-examples: someone who's an outlier on hours per week spent exercising is less likely to be an outlier on blood pressure. Any type of dependence is possible in principle—Chaos Theory doesn't say the butterfly that flaps its wings hardest causes the biggest hurricane (I don't know what Zen Buddhism says). A general theory that ranges over all measurements anyone might want to make on all kinds of things is going to be elusive.

If you consider a world with a well-defined set of objects, $a_1, a_2, \ldots$, & a well-defined set of measurements we can make on each object, $x_{11}, x_{12}, \ldots, x_{21}, x_{22}\ldots$, then your question could perhaps be framed in an answerable way ("Pick $n$ objects at random from the $a$s, pick two measurements at random from the $x$s, ..."); for our world I don't see how it can be.

• I've no idea how one could even formulate the notion in such a way that it could be empirically proved or disproved. Nor of in what circumstances it might be rational to allow it to over-ride particular prior knowledge. – Scortchi - Reinstate Monica Oct 24 '13 at 12:57
• It's a counter-example only to the idea that, for any given probability model of the dependence between two measurements, an individual's being extreme in one increases the probability of its being extreme in the other. @Glen pointed out that that isn't necessarily the case, & I was only trying to show that its not implausible in practice that it isn't. (I'm not saying it isn't obvious, or that you thought otherwise, but it's worth pointing out in response to your question as originally written.) – Scortchi - Reinstate Monica Oct 27 '13 at 20:30
• As I've tried to say, your conjecture that measurements on things are generally correlated in a particular way is too vaguely specified to be able to talk about it in terms of probability. You'd have to specify the universe of "measurements" & of "things" you're interested in. – Scortchi - Reinstate Monica Oct 27 '13 at 20:35
• @Behacad: Blood pressure doesn't go down more & more as you exercise more & more - someone who exercises a lot is especially likely to have bp in the normal range (unless we're considering a terribly unfit population where hypertension is the norm). More importantly, would you care to define what this "general probability" in fact means? – Scortchi - Reinstate Monica Oct 30 '13 at 17:50
• I clearly lack the type/level of intuition needed to understand what's meant by a weak tendency of nature that doesn't exist in a measurable way. I'll stick to my statistical last & leave the more highly-evolved souls to contemplate it in peace. – Scortchi - Reinstate Monica Oct 31 '13 at 10:00

I'm going to say yes. Of course something being an outlier on one measure does not necessarily mean it will be an outlier on other measures, but I think it certainly increases the odds. In essence, I think anything being unusual in one way is more likely to be unusual in other ways. This is perhaps the result of different variables being correlated in all kinds of ways. So if one is "extreme", this will also correlate with other factors that might also be in the extreme range.

I hope someone can provide a better answer with some empirical evidence. Perhaps we can look at this from the multivariate outlier perspective. In my experience, individuals who are outliers on one variable are often outliers on many other variables as well, contributing to multivariate outlierness.

Take this answer with a grain of salt since it depends on how we define an outlier and such, but I think in general it makes sense.

• You've definitely added to my question and also helped to refine it - I was also mostly thinking about this in terms of individuals/choices/personality traits/etc. but I don't think that it stops there. I suppose I'm looking for some sort of theorem or principle that attempts to explain this elegantly. Michael Mayer did have a good response though. – Taal Oct 23 '13 at 19:29

I will try to reply using empirical evidence. Let's assume you are measuring the heights in a men sample. In this case, the outlier will be represented by a very tall men (a giant). It is very likely this men will represent an outlier also for other variables like for instance shoe size or arms lengths and so on. Other case, you are measuring financial performance of US Public company. An outlier will be a very successful company with a sales growth twice the industry average. Very likely the same company will be an outlier in respect of any measure of profitability or stock price appreciation. In a nut shell, I am incline to think something behaving exceptionally out of the norm will tend to conserve this property across different manifestations. Is there a theorem that disprove this theory?

• Yes, another good representation of what I'm trying to communicate - and yes also looking for a theorum that speaks about this as my ultimate question (I need to edit the original actually). As another example, I was thinking about the personalities/etc. of girls that "dye their hair blue." This is not a very usual thing to do where I live, so I'm thinking it would be safe to assume there is a higher probability (in comparison to girls that didn't have an outlier as such...and yes I'm controlling the experiment here a bit) that this girl will also have other "outlier-ish" characteristics. – Taal Oct 23 '13 at 22:44
• The "sales growth" example is somewhat flawed - many extremely high sales growth companies are unprofitable (see: Amazon). But more importantly, all of your examples are dependent on each other. Then what you're describing isn't a property of "outlier-ness" but on the value itself. Saying a tall person is likely to have high values for characteristics associated with height is just correlation. – Fomite Oct 30 '13 at 18:17
• @EpiGrad The conjecture is that this is a weak tendency for a probability to be higher (and the degree of "higher" may even be quite small...but still exist in nature. (I'm realizing I really should probably update the question a bit now). I believe the poster here understood this and assumed, like I did, that people wouldn't interpret the examples in the fashion that this is true when the variables are ONLY "dependent" on each other in such an obvious way. Basically, one must agree that everything in the universe is dependently related to everything else - and this really is true. – Taal Oct 31 '13 at 9:19
• TRUE "independent" variables simply do not exist - really think about that. We simply assume and throw the label onto them that they are "independent" to be able to use methods that have been created in statistics in a practical manner. – Taal Oct 31 '13 at 9:21
• @Taal Given I've encountered variables that are about as independent as they come (seriously, once found a relative risk of 1.00 with a 95% confidence interval from 0.99 to 1.01), I'm going to have to suggest "Citation Needed". And if the poster, and you, aren't relying on variables that are only dependent in obvious ways, I'd suggest finding concrete examples of that, rather than simply asserting they exist. – Fomite Oct 31 '13 at 18:35

Outliers are a reflection of an unknown/unspecified external factor. If there is a relationship between two series then there would be an increased probability that both series would be affected. My answer to your question is "yes" since there may be a relationship between the two series.

• I like the first sentence in your answer alot. When you say "if there is a relationship between two series"...it makes me strongly believe you are, like many others have (but not as intensly as they have), jumping ahead a slight bit from where I'm starting - which does not look for a relationship before assuming the conjecture is true. The conjecture is actually the tool in which one would use to potentially discover new relationships - it comes before assuming any relationships besides the fact that all the observations (hate that word here actually) - are the same..cont.. – Taal Oct 31 '13 at 8:40
• ...cont...example: All the observations are measuring or looking at something involving blades of grass. And actually - really - to be honest it assumings every things/measurement/idea/noun/etc. is related to everything else no matter how weak that relationship may be. It may be so weak it's essentially undetectable - and this comes from the fact that the use of the words "dependent" or "indepdent" are actually misnomers - but must be used (for practicality's sake) due to the limits of current human knowledge in math/statistics. – Taal Oct 31 '13 at 8:46

The answer, in my mind, is "Yes and No, and if Yes then this isn't actually interesting".

For variables with no dependency structure, the answer is no - a very high, or very low, value of a particular variable doesn't imply a very high, or very low value of another variable.

For variables with a dependency structure, the answer is often (but not always) yes. But this isn't a property of "outlier-ness", it's a property of correlation itself. What you're showing is not that "Being an outlier begets being an outlier in other areas" but that two associated variables behave exactly like associated variables should.

In another question I left an answer to, @FrankHarrell left a separate answer that I to this day still find quite profound.

He says "Variable selection without penalization is invalid."

If you're attempting to analyze any set of data and determine the influence of variables on each other, then, really, the only variables you're using are the ones you have in your dataset - even if you discover that there is no relationship between certain variables.

Let's say X(sub 5) has what seems to be absolutely no relationship to your Y - so you discard it.

If you discard it, though, then you're selecting variables...incurring penalization. Additionally, if you're missing ANY variables in your dataset, you're selecting variables once again. When I say ANY, I literally mean any and every variable in the universe.

And, well, "Variable selection without penalization is invalid."

Thus, if

"Variable selection without penalization is invalid" is true,

then the statement

All variables in the universe are related/influence each other/are affected by each other/etc. must be true.

Otherwise, penalization would not occur with variable selection.

If I were doing a binomial test, and I had 100% of my results indicating pass until the nth sample, at which time I had one sample indicate fail, then I would say that my estimate of the binomial distribution parameters and their confidence intervals might be very different before versus after the "fail".

My binomial test is whether or not the samples belong to an expected distribution which if true gets a "pass" or whether they qualify as an outlier in which case the result is "fail".

Keywords of interest may include "zero defect sampling", and "acceptance sampling".