3
$\begingroup$

I have been using several techniques, sometimes in conjunction with one another, in a pre-processing pipeline prior to using the data for supervised machine learning.

I was wondering about the effects of the order that I perform the operations.

Several operations that I have been performing are:

Outlier removal (either removing all instances that contain any feature with a value 3 standard deviations above or below the mean, or windsorizing the feature to the corresponding value)

Downsampling (I either remove instances from the majority class until they equal the minority class or upsample the minority class using SMOTE)

Standardization (Using the z-score for a feature instead of the raw score)

Standardization and outlier removal are done on a within subject basis.

I feel like outlier removal should go before standardization, as the standardization would be effected by any outliers, but outlier removal would still work fine after standardization (since I am computing the z-score to remove outliers anyway)

Downsampling is currently performed last.

Could anyone explain some of the interactions between these, or point me to somewhere that would have some of this information?

|cite|improve this question
$\endgroup$
4
$\begingroup$

Remove outliers, then standardise. This way all the batches of your "good data" will be scaled consistently.

Downsampling, as in removing data points, seems rather sketchy. If required, you could always just do stratified sampling instead etc.

@Automated pipelines You have too many outliers, remove them and you remove an important chunk of the dataset. Or, those outliers were really important, and suddenly the predictions are bad.

I find using automated pre-processing of features about as feasible as flying planes without a pilot. You can set up the pipeline, but if the output really matters, you will always have to check yourself.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thanks for your comment! Downsampling is only ever done on the training set. The way we see it, we can train the model on whatever we want, but then the testing data (which is completely subject independent) indeed does not have downsampling. We use downsampling because we have a skewed class distribution. I'm not sure if stratified sampling would help, because we don't want to retain the class distribution. $\endgroup$ – rbx Jan 14 '14 at 1:24
  • 1
    $\begingroup$ Certainly, you can train on whatever you want. But are you having a lot of success with training on a symmetric distribution of the target variable and predicting for a skewed one? If so, it seems like an exceptional situation then - would love to see a counter-example on this. In general, I would want the training data to be representative of the testing dataset, across all variables, as much as possible. Regarding the point with the stratified sampling, I was thinking about it as a way how to deal with skewness during prediction, for example if you do cross-validation for parameter tuning. $\endgroup$ – means-to-meaning Jan 14 '14 at 23:04
  • $\begingroup$ For some time series data downsampling might be inevitable, and then it should certainly be before the actual standardisation, because otherwise the dimensions won't match. $\endgroup$ – boomkin Jul 3 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.