I'm unsure on how to remove or winsorize outliers. Let's say I have 2 groups, treated and control. And I measure feature1 and feature2 for both.
How should I handle outliers? For each group and each feature separately? Meaning: Feature 1 group 1, feature 1 group 2, feature 2 group 1 and feature 2 group 2
Or as a whole group?
First, you shouldn't remove or winsorize outliers unless they are data entry errors. Data entry errors should be corrected (if possible) or removed.
Second, you can try to identify data entry errors by feature; it doesn't make sense to combine the features into one. However, you may look for bivariate outliers. I would do this graphically using boxplots, quantile normal plots, density plots, and so on. I would do this by group, as what is an outlier in one group may not be in another. (Just as an extreme example, if one feature is height and the groups are jockeys and basketball players).
You can also look at the output from your ANOVA. The major packages provide plots for examining problems (certainly R and SAS do, but I can't imagine that SPSS etc. don't do this).
Third, if you do identify outliers or influential points, then change your method, not your data. Two ideas are robust regression (there are several varieties) or quantile regression. Some people will recommend transforming your data, but I think this is only a good idea when the transformation makes substantive sense (opinions differ on this).
