0
$\begingroup$

I am currently working on my bachelor thesis in finance and I faced some problems regarding my dataset. I wanted to analyze the effect of leverage on the performance of companies and as many researchers before me, I wanted to use a multiple linear regression analysis. My tutor advised me to winsorize the data at 2.5% and 97.5%. However, when I checked the statistics for it, for some of my variables over 200 observations out of 4000 have been detected as outliers. For other variables even 2000 observations are being marked as outliers. I was searching for answers on the web and tried different methods in order to reduce the numbers. However, it still doesn’t work, and it doesn’t make any sense to me. It would be more than lovely if someone could give me some advice on it.

Thank you :)

$\endgroup$
4
  • $\begingroup$ How are you defining "outlier" in this context? $\endgroup$ – The Laconic Jun 9 '18 at 22:28
  • $\begingroup$ I conducted a 97.5% confidence interval and as I thought the values above 97.5% and below 2.5% are extreme values and in this case outliers $\endgroup$ – Christoph Jun 9 '18 at 22:56
  • $\begingroup$ I am sorry if this was wrongly used. I mean even though we had some statistic courses we never discussed the method of winsorizing data. What I did was that I used the explore function in SPSS, constructed a 97,5% confidence interval of the mean. Then, I received some descriptive statistics and I thought the values below the lower bound and the values above the upper bound are the outliers. Maybe I am completely wrong but that was the way I thought I have to do it. $\endgroup$ – Christoph Jun 10 '18 at 16:15
  • $\begingroup$ @Christoph , It's not entirely clear to me what you did, but just for posterity, it doesn't sound like a reasonable approach to identify outliers. $\endgroup$ – Sal Mangiafico Jun 10 '18 at 17:30
4
$\begingroup$

If you have 4000 observations and you winsorize the top 2.5% and bottom 2.5% of data, then 200 observations will be affected. It doesn't matter what these values are, and it doesn't imply that they were outliers in any meaningful sense of the term.

Winsorizing data shouldn't remove any observations, but it will change them.


EDIT: Some additional information in response to comments.

One distinction to make is between trimming and Winsorization. Trimming will simply remove observations that fall outside of specified quantiles. So trimming to 95% will remove the top 2.5% of observations and the bottom 2.5% of observations.

Winsorizing doesn't remove observations, but changes the values of those observations outside a specified quantile to the value at that quantile. I think this makes sense with a simple example.

One word of caution is that there are different methods to find percentiles, so the defaults on other software packages may find somewhat different results.

Here, the data are Winsorized to 60%. The 20th percentile is calculated as 2.8 and the 80th percentile is calculated as 8.2. So the values less than 2.8 are replaced by 2.8 and the values greater than 8.2 are replaced with 8.2.

if(!require(psych)){install.packages("psych")}

A = c(1,2,3,4,5,6,7,8,9,10)

quantile (A, c(0.20, 0.80))

   ### 20% 80% 
   ### 2.8 8.2 

library(psych)

winsor(A, trim = 0.20)   # This Winsorizes to the inner 60% of observations

###   [1] 2.8 2.8 3.0 4.0 5.0 6.0 7.0 8.0 8.2 8.2
$\endgroup$
8
  • $\begingroup$ It's surely prudent to keep the original data as well as any Winsorized version of them. $\endgroup$ – Nick Cox Jun 10 '18 at 15:16
  • 1
    $\begingroup$ There's a whole other discussion about whether Winsorization is justified, or, in this particular case, desirable. I would advise against routinely changing or dropping data. And also remind that sometimes the extreme values are the interesting values. But without the specifics of the goals of the analysis, it's difficult to know what effect (good, bad, ugly) Winsorizing may have. $\endgroup$ – Sal Mangiafico Jun 10 '18 at 16:08
  • 1
    $\begingroup$ As @SalMangiafico has already pointed out: the central $p$% of the data are not at all the same as a $p$% confidence interval for the mean. Values outside that central part are usually not outliers in any other sense. I guess that almost no introductory course or text anywhere mentions Winsorizing (an exception is the excellent text of Dixon and Massey, 4th edition 1983). $\endgroup$ – Nick Cox Jun 10 '18 at 16:47
  • 1
    $\begingroup$ Oh my gosh! I was troubling with this issue for three days and a few colleagues of mine as well. Thank you very very much! :D $\endgroup$ – Christoph Jun 10 '18 at 17:12
  • 2
    $\begingroup$ The median after Winsorizing is the same as before it! $\endgroup$ – Nick Cox Jun 11 '18 at 10:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.