Questions tagged [winsorizing]

Winsorizing is a kind of data transformation used in robust/resistant statistics. Extreme values in the sample is replaced by some chosen data quantile(s). See https://en.wikipedia.org/wiki/Winsorizing

Filter by
Sorted by
Tagged with
8
votes
1answer
1k views

Downweight outliers in mean

I have a bunch of points $x_i$ and would like to calculate a kind of weighted mean that deemphasizes outliers. My first idea was to weight each point by $1/ (x_i - \mu)^2$. However, the problem is ...
32
votes
5answers
22k views

What are the relative merits of Winsorizing vs. Trimming data?

Winsorizing data means to replace the extreme values of a data set with a certain percentile value from each end, while Trimming or Truncating involves removing those extreme values. I always see ...
1
vote
1answer
42 views

Winsorizing propensity scores

Is it kosher? Inverse propensity weights (IPW) has been shown to perform poorly when selection probabilities are small (Kang and Schafer, 2007). Are there any standard solutions to this issue?
0
votes
1answer
301 views

functional differences between using huber loss and winsorizing/trimming

Curious what the functional differences are between using a Huber loss function/ regression and Winsorizing data and then running a classic least squares regression. Will the resulting outputs be ...
1
vote
2answers
725 views

What is the difference between GAS ( Generalized Autoregressive Score) model and a GARCH?

I am trying to analyze some data about Brent Oil volatility. So far I have managed to fit a GARCH(1,1) model and an EGARCH. However, someone has recommended to use a GAS model, Generalized ...
19
votes
4answers
61k views

Should the mean be used when data are skewed?

Often introductory applied statistics texts distinguish the mean from the median (often in the the context of descriptive statistics and motivating the summarization of central tendency using the mean,...
1
vote
0answers
9 views

Name for the opposite of Winsorizing?

For some regressions we find it useful to focus on extreme values, and so we discard middling dependent values (which we might call "noise") from data in order to find relationships that hold at data ...
4
votes
1answer
2k views

use and misuse of Winsorization

I am doing research on Winsorization (and trimming), which has been broadly applied in many fields, but I think many researchers didn't do it in a "rigorous" way. Or maybe even worse, they misuse it. ...
0
votes
2answers
339 views

Is Winsorization performed on test data as well?

I know what is Winsorization and why is it applied. My understanding was that it is applied only on the train data to reduce the effect of outliers. But! Recently I came across a kernel where Min, ...
33
votes
8answers
28k views

Replacing outliers with mean

This question was asked by my friend who is not internet savvy. I've no statistics background and I've been searching around internet for this question. The question is : is it possible to replace ...
19
votes
5answers
9k views

Which robust correlation methods are actually used?

I plan to do a simulation study where I compare the performance of several robust correlation techniques with different distributions (skewed, with outliers, etc.). With robust, I mean the ideal case ...
0
votes
0answers
208 views

How to choose cut off for winsorization/ flooring- capping? What is the impact of variable distribution on the decision

To perform logistic regression I wish to winsorize outliers in independent/ explanatory variables by flooring and capping independent variables. Can you suggest how I should choose cut-off for ...
0
votes
0answers
222 views

Winsorizing data in small sample

I have a relatively small sample of panel data (quarterly data for 68 firms over 7 years). My dependent variable is positively skewed. In order to limit the influence of observations with large values,...
4
votes
1answer
863 views

Linear regression with violated assumptions

I am trying to find out the determinants of cognitive function. The outcome variable is the mini–mental state examination which is a 30 point questionnaire response that has score values from 0 to 30(...
6
votes
3answers
4k views

Extreme values in the data

I have a very general statistical question. If a variable has some extreme values, then for the purpose of statistical inferences for example OLS regression, is it better to detect these extreme ...
1
vote
1answer
455 views

Greater than 30% outliers in small dataset - what to do? Standard test? Test with outliers removed? Robust statistics?

I have a small-sample dataset representing observations from a longitudinal study. My principal interest is in 'change scores' across three parameters (A, B, C). This requires simple paired t-tests. ...
0
votes
1answer
2k views

Winsorizing data

I am currently working on my bachelor thesis in finance and I faced some problems regarding my dataset. I wanted to analyze the effect of leverage on the performance of companies and as many ...
3
votes
0answers
924 views

Treating outliers for time series forecasting in Python

What is the best way to treat outliers in a time series forecasting model? In particular, for product demand modeling? Based on what I've read so far, the following methods can be applied: ...
0
votes
0answers
354 views

Winsorization to remove spikes in time series

In product demand forecasting, is it valid to use winsorization to remove large outliers (spikes) in the data? I understand that the spikes may be due to holiday effects (e.g. people will buy more ...
1
vote
1answer
1k views

Treatment of outliers in financial data

I have a data set with financial panel data from 150 companies. I want to analyse the data using linear repeated measures ANOVA and OLS Regression (so far). For this, I want to use the absolute values ...
2
votes
0answers
161 views

Does pre winsorising of a variable help for a logistic regression?

I am wondering if winsorising makes a difference in a logistic regression. In a situation where I am looking at the individual contribution, looking at their individual discriminatory power (...
0
votes
1answer
34 views

winsoring forecasting dataset

I have performed a logistic regression to estimate the default probability of a dataset of firms based on some basic balance-sheet ratios. I have winsorized all the ratios at the 1st and 99th ...
2
votes
2answers
593 views

Removing outliers and calculating a “lowest” attainable price from a pre-determined/fixed time series of prices

Just a foreword, I'm not a mathematician or otherwise statistically skilled. I know my way around calculating standard deviations, but it's all self taught. I'm a programmer with limited stats ...
4
votes
3answers
335 views

In a “bursty” dataset, how do you filter for the few important values that make up the bulk of the information?

Note sure if there is an existing stats concept for this but I have a dataset that consists of mostly small data points with a few large ones. e.g. 1 2 1 3 1 2 87 3 2 1 1 1 1 3 1 2 1 1 1 99 How can ...
2
votes
1answer
417 views

Ensemble time series prediction from two separate models

I have two different forecasts that are produced by ARMA models using two different data samples. The difference between the two data sets is their size: one used data from 2013-2014 and another used ...
8
votes
2answers
3k views

Alternatives to using Coefficient of Variation to summarize a set of parameter distributions?

Background I have a model with 17 parameters, and I currently use the coefficient of variation ($\text{CV}=\sigma/\mu$) to summarize the prior and posterior distributions of each parameter. All of ...
2
votes
1answer
428 views

Combining similarity scores

I have a list of m x n similarity score matrix, something like ...
3
votes
2answers
506 views

How to best estimate the time remaining for a variable-length questionnaire?

The greatest gain of the statistics classes in both school and university seems to be that I now have an inkling of which QA site to use for this question. :) I'm a programmer and I'm making a ...
2
votes
1answer
742 views

Scale independent forecast error metric that works with changing signs

I am trying to analyze a quite large (~25,000 rows) dataset of cash flow forecasts. Receipts and expenses are aggregated, thus I may end up with the following data: ...
11
votes
5answers
22k views

How to correct outliers once detected for time series data forecasting?

I'm trying to find a way of correcting outliers once I find/detect them in time series data. Some methods, like nnetar in R, give some errors for time series with big/large outliers. I already managed ...
4
votes
3answers
2k views

How to describe the differences in skewed data with same median but statistically different distribution?

I am comparing length of stay after laparoscopic and open appendectomy in over 160000 patients. LOS is typically a skewed variable so I use the median and interquartile range and ranksum test to ...
0
votes
1answer
108 views

How to evaluate a curve considering outliers?

I have data on runners who run marathons; for each runner I have their final times on a number of races. I would like to predict how fast they are running considering outliers i.e. he's running ...
1
vote
2answers
806 views

How do I calculate the ranking of some galleries based on the rankings of the artists represented by them?

The mean is not good in this case, because there are galleries that have an artist with a high rank and several other artists with way lower ranks. I'm thinking about doing a weighted mean, but I don'...
2
votes
2answers
2k views

Measure of closeness

Given a list of numbers, is it possible to find out (or in other words, is there a statistical measure to tells the) the closeness of the numbers (do note that i am not talking about correlation - ...
2
votes
3answers
2k views

Robust standardization of data

I have some data where I want to determine whether the shape of the probability distribution has changed compared to 10 years ago. One example is that I have for various automobiles multiple measures ...
2
votes
2answers
229 views

Multilevel modeling for limited dependent variable

I am doing the research, using Multilevel modeling, with limited dependent variable number of days- it is limited downward (0) and upward (30). Is it necessary to use Multilevel logit model? Or is it ...
6
votes
2answers
450 views

How to average quantized and truncated data?

So I have data that has been quantized by an analogue to digital converter. (continuous data has been turned into discrete data and the values range from 0 to the saturation value , which is 127 in ...
7
votes
2answers
2k views

Outlier detection for generic time series

In this case, "generic" being the entire gauntlet of macroeconomic time-series that private and government statistical offices put out. Some background - I recently started working at a data provider ...
3
votes
3answers
2k views

Removing outliers from asymmetric data

I have a data set that includes the number of visits to a website. Here are some descriptive statistics for my data Median: 4 Mean: 14.1352 SD: 121.8119 Clearly, there are some huge values (...
1
vote
1answer
640 views

Measure of central tendency for an ex-gaussian distribution

I know there won't be a clear answer to that question but I'm really curious to know your opinion on that matter. I deal with reaction times, and finding a good measure of central tendency is ...
2
votes
0answers
110 views

Optimizing Robust Statistics

A robust paired t-test is a better choice for skewed distributions than the conventional paired t-test (e.g Fradrette, Keselman, Lix, & Wilcox, 2003). One version of the robust test uses a trimmed ...
4
votes
2answers
785 views

Dealing with outliers when comparing variances with Bartlett's test

I have four different groups (with unequal sample sizes of 100 to 120) and want to test if the variance differs. For ANOVAs I used the winsorized mean to get a more robust estimate and I am wondering ...
7
votes
1answer
2k views

Does Pearson correlation require removal of bivariate or univariate outliers?

Does the Pearson's correlation estimator require no bivariate outliers, or no outliers in each of two individual vectors of data? The answer will impact on how I winsorize outliers before calculating ...
3
votes
3answers
79 views

Limiting the range of numbers

Suppose that I have the following data set: {0.1, 0.2, 0.5, -0.1, 0.5, 1.1, 0.8} I would like to limit the range of these data to be within the range of [0,1]. ...
2
votes
1answer
502 views

Does the Hodges-Lehmann estimator perform better than trimmed/winsorized means?

I've been reading about the HL estimator, and a question came to mind. I could fairly easily create a mean-estimator where I trim or clip 29% of the data on either side and have a statistic with a ...
0
votes
0answers
394 views

Using trimmed means and Winsorized variances to compute standardisation of data

I am looking into the pros and cons of each normalisation technique for work and it got me thinking. What if I used trimmed means and the sqrt of Winsorized variances to compute the standardised data? ...
2
votes
0answers
179 views

Robust Estimators - Winsorized Variance degree of freedom (df)

this is my first question on this site. So, I'm currently working on my final year thesis, and it was on Robust statistics. In my work, I will use Trimmed Mean, Winsorized Mean and Winsorized ...
1
vote
0answers
137 views

Winsorization when we run regressions by size group

I have a sample that consists of large, medium, and small firms and i want to run a separate regression for each size group. When I winsorize a variable should I do it for the whole sample (i.e. ...
5
votes
1answer
550 views

Why are Winsorized random variables independent?

While studying trimmed mean I understood that if I have some random variables $X_1, X_2, .., X_n$ by ordering them and trimming, the variables are no longer independent. However it is said that "by ...
1
vote
0answers
81 views

Winsorizing, just the outlier or all the value?

I have an outlier in my data set. I want to use the winsorizing quartile (to change the outlier to the 5th% and/or 95th%). Looking at the quartiles, sometimes I have more values than just my ...