This question was asked by my friend who is not internet savvy. I've no statistics background and I've been searching around internet for this question.
The question is : is it possible to replace outliers with mean value? if it's possible, is there any book reference/journals to backup this statement?
Clearly it's possible, but it's not clear that it could ever be a good idea.
Let's spell out several ways in which this is a limited or deficient solution:
In effect you are saying that the outlier value is completely untrustworthy, to the extent that your only possible guess is that the value should be the mean. If that's what you think, it is likely to be more honest just to omit the observation in question, as evidently you don't have enough information to make a better guess.
With nothing else said, you need a criterion or criteria for identifying outliers in the first place (as implied by @Frank Harrell). Otherwise this is an arbitrary and subjective procedure, even if it is defended as a matter of judgment. With some criteria, it is possible that removing outliers in this way creates yet more outliers as a side-effect. An example could be that outliers are more than so many standard deviations away from the mean. Removing an outlier changes the standard deviation, and new data points may now qualify, and so on.
Presumably the mean here means the mean of all the other values, a point made explicit by @David Marx. The idea is ambiguous without this stipulation.
Using the mean may seem a safe or conservative procedure, but changing a value to the mean will change almost every other statistic, including measures of level, scale and shape and indicators of their uncertainty, a point emphasized by @whuber.
The mean may not even be a feasible value: simple examples are when values are integers, but typically the mean isn't an integer.
Even with the idea that using a summary measure is a cautious thing to do, using the mean rather than the median or any other measure needs some justification.
Whenever there are other variables, modifying the value of one variable without reference to others may make a data point anomalous in other senses.
What to do with outliers is an open and very difficult question. Loosely, different solutions and strategies have varying appeal.
As a very broad-brush generalisation, there is a continuum of views on outliers in statistics and machine learning from extreme pessimists to extreme optimists. Extreme pessimists feel called to serve as if officers of a Statistical Inquisition, whose duty it is to find outliers as obnoxious contaminants in the data and to deal with them severely. This could be the position, say, of people dealing with financial transactions data, most honest or genuine, but some fraudulent or criminal. Extreme optimists know that outliers are likely, and usually genuine -- the Amazon, or Amazon, is real enough, and really big. Indeed, outliers are often interesting and important and instructive. Floods, fires, and financial crises are what they are, and some are very big.
Here is a partial list of possibilities. The ordering is arbitrary and not meant to convey any order in terms of applicability, importance or any other criterion. Nor are these approaches mutually exclusive.
One (in my view good) definition is that "[o]utliers are sample values that cause surprise in relation to the majority of the sample" (W.N. Venables and B.D. Ripley. 2002. Modern Applied Statistics with S. New York: Springer, p.119). However, surprise is in the mind of the beholder and is dependent on some tacit or explicit model of the data. There may be another model under which the outlier is not surprising at all, so the data really are (say) lognormal or gamma rather than normal. In short, be prepared to (re)consider your model.
Go into the laboratory or the field and do the measurement again. Often this is not practicable, but it would seem standard in several sciences.
Test whether outliers are genuine. Most of the tests look pretty contrived to me, but you might find one that you can believe fits your situation. Irrational faith that a test is appropriate is always needed to apply a test that is then presented as quintessentially rational.
Throw them out as a matter of judgement.
Throw them out using some more-or-less automated (usually not "objective") rule.
Ignore them, partially or completely. This could be formal (e.g. trimming) or just a matter of leaving them in the dataset, but omitting them from analyses as too hot to handle.
Pull them in using some kind of adjustment, e.g. Winsorizing.
Downplay them by using some other robust estimation method.
Downplay them by working on a transformed scale.
Downplay them by using a non-identity link function.
Accommodate them by fitting some appropriate fat-, long-, or heavy-tailed distribution, without or with predictors.
Accommodate by using an indicator or dummy variable as an extra predictor in a model.
Side-step the issue by using some non-parametric (e.g. rank-based) procedure.
Get a handle on the implied uncertainty using bootstrapping, jackknifing or permutation-based procedure.
Edit to replace an outlier with some more likely value, based on deterministic logic. "An 18- year-old grandmother is unlikely, but the person in question was born in 1932, and it's now 2013, so presumably is really 81."
Edit to replace an impossible or implausible outlier using some imputation method that is currently acceptable not-quite-white magic.
Analyse with and without, and seeing how much difference the outlier(s) make(s), statistically, scientifically or practically.
Something Bayesian. My prior ignorance of quite what forbids from giving any details.
EDIT This second edition benefits from other answers and comments. I've tried to flag my sources of inspiration.
There are several problems implied by your question.
None of 1-5 have an obvious answer. If you really feel that these "outliers" are wrong and you don't want to use a robust statistical method, you can make them missing and use multiple imputation as one possible solution. If the variable is a dependent variable, one robust choice is ordinal regression.
The proposal has numerous flaws in it. Here is perhaps the biggest.
Suppose you are gathering data, and you see these values:
$$2, 3, 1$$
The mean, so far is $6/3 = 2$.
Then comes an outlier:
$$2, 3, 1, 1000$$
So you replace it with the mean:
$$2, 3, 1, 2$$
The next number is good:
$$2, 3, 1, 2, 7$$
Now the mean is 3. Wait a minute, the mean is now 3, but we replaced 1000 with a mean of 2, just because it occurred as the fourth value. What if we change the order of the samples?
$$2, 3, 1, 7, 1000$$
Now the mean prior to the 1000 is $(2 + 3 + 1 + 7)/4 = 13/4$. So should we replace 1000 with that mean?
The problem is that the false datum we are substituting in place of 1000 is dependent on the other data. That's an epistemological problem if the samples are supposed to represent independent measurements.
Then you have the obvious problem that you not merely withholding data that doesn't fit your assumptions, but you're falsifying it. When some unwanted result occurs, you increment $n$, and substitute a fake value. This is wrong because $n$ is supposed to be the count of samples. Now $n$ represents the number of samples, plus the number of fudge values added to the data. It basically destroys the validity of all calculations involving $n$: even those which do not use the fudge values. Your $n$ is a fudge value too!
Basically, trimming away results that don't fit is one thing (and can be justified if it is done consistently according to an algorithm, rather than according to changing mood swings of the experimenter).
Outright falsifying results is objectionable on philosophical, epistemological and ethical grounds.
There may be some extenuating circumstances, which have to do with how the results are used. Like for instance, say that this substitution of outliers by the current mean is part of some embedded computer's algorithm, which enables it to implement a closed-loop control system. (It samples some system outputs, then adjusts inputs in order to achieve control.) Everything is real time, and so something must be supplied for a given time period in the place of missing data. If this fudging helps to overcome glitches, and ensures smooth operation, then all is good.
Here is another example, from digital telephony: PLC (packet loss concealment). Crap happens, and packets get lost, yet communication is real time. PLC synthesizes fake pieces of voice based on recent pitch information from correctly received packets. So if a speaker was saying the vowel "aaa" and then a packet was lost, PLC can pad the missing packet by extrapolating the "aaa" for the frame duration (say 5 or 10 milliseconds or whatever). The "aaa" is such that it resembles the speaker's voice. This is analogous to using a "mean" to substitute for values regarded as bad. It's a good thing; it's better than the sound cutting in and out, and helps intelligibility.
If the fudging of data is part of a program of lying to people to cover up failing work, that's something else.
So, we cannot think about it independently of the application: how is the statistics being used? Will substitutions lead to invalid conclusions? Are there ethical implications?
This article by Cousineau and Chartier discusses replacing outliers with the mean
http://www.redalyc.org/pdf/2990/299023509004.pdf
They write:
Tabachnick and Fidell (2007) suggested replacing the missing data with the mean of the remaining data in the corresponding cell. However, this procedure will tend to reduce the spread of the population, make the observed distribution more leptokurtic, and possibly increase the likelihood of a type-I error. A more elaborate technique, multiple imputations, involves replacing outliers (or missing data) with possible values (Elliott & Stettler, 2007; Serfling & Dang, 2009).
There is also an R package "outliers" that has a function to replace outliers with the mean. I also saw a number of hits in my Google search that implies that SPSS also has such a function, but I am not familiar with that program. Perhaps if you follow the threads you can discover the technical basis for the practice.
The main thing to bear in mind when dealing with outliers is whether they're providing useful information. If you expect them to occur on a regular basis then stripping them out of the data will guarantee that your model will never predict them. Of course, it depends what you want the model to do but it's worth bearing in mind that you shouldn't necessarily drop them. If they contain important information you may want to consider a model that can account for them. One, simple way to do that is to take logs of the variables, which can account for power law relationships. Alternatively, you could use a model that accounts for them with a fat-tailed distribution of errors.
If you do want to cut them out then the usual ways are to either drop them or Winsorise them to remove the extreme values. I don't have a textbook to hand but the Wiki links there do refer to some if you want to read further. Most texts on applied statistics should have a section on outliers.
I'm aware of two related similar approaches in statistics.
For more detailed examples, see Wikipedia:
https://en.wikipedia.org/wiki/Trimmed_estimator
https://en.wikipedia.org/wiki/Winsorising
Note that this works good for some statistics such as when computing the mean. The trimmed / winsorized mean is often a better estimate of the true mean than the artihmetic average. In other cases, it may ruin your statistics. For example when computing variance, trimming will always underestimate your true variance. Winsorization, assuming that indeed some of the extreme observations are faulty, will work a bit better then (it will probably still underestimate, but not by as much).
I don't see how replacing the extreme values with the mean would fit in here.
However, there is another practice that is related: missing value imputation. Assuming that your outlier is flawed, worthless data, so your remove it. When you then perform imputation, a typical substitute value would be the mean or mode:
The traditional approach for handling outliers is to simply remove them such that your model is trained only on "good" data.
Keep in mind that the mean value is affected by the presence of those outliers. If you replace outliers with the mean calculated after the outliers were removed from your dataset, it will make no difference since the regression line (from simple linear regression) will pass through the mean of your training data anyway (this will reduce the variance of your estimates though, which is probably the opposite of what you want given that you know there are outliers).
The effect your approach will have on the model depends on the influence (leverage) of the outlier. I'd recommend against the approach you suggest in lieu of just removing the point entirely.
yes the outliers can be replaced in may forms, for example, let's take a data-set of the size of Human heights, let's say we have some outliers like 500 cm and 400 cm then, we can just replace those data points that appear in the dataset because of some error that was caused during the recording of the data. so the options you can try is 1. replace it with the Median of the Whole color of the data (not the mean, as it is prone to outliers). 2. replace with the most Occurring data point in the Column. 3. If Categorial values then you can try Response coding.(wherein you Record the Probability of the word or the values occurring by the total number of words )
