1
$\begingroup$

Two questions here :

  1. Should transformation of skewed distribution is neccessary for classification problem ?
  2. And consider a case where I have already transformed variables , so the equation will work for the transformed variables. Now when I use this model on a new and never seen data where the variables are not transformed , will it hold good ?
Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$
  1. From my experience it's not necessary, but transforming variables can be very beneficial. It's also always judgement call depending on the variable. It's also dependent on the type of model you're using; for logistic regression it's a good idea and one recommendation would be to use Weight of Evidence (WOE) binning.

For other tree based algorithms, such as Random Forest, you don't need to as the algorithm will make the cuts itself.

  1. The answer is No. For the sake of argument, let's say you have create a Logistic Regression, with one transformed variable.

That transformed variable is actually a NEW variable, so once you get a new dataset, you should apply the same transformations and then make your predictions.

If I haven't been clear in any of my points just let me know and I will be happy to elaborate. :)

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you Ehsan. I was wondering about Kaggle Data science competitions . Many competitions have test set with them. So if I apply transformations on my training and validation set, it won't rather can't be applied to test set as they are not accessible. Any idea or suggestion ? $\endgroup$
    – Mighty
    Commented Nov 8, 2017 at 12:48
  • $\begingroup$ I think I understand what you mean. So in this case, your code structure should be: 1) Transform variables, 2) Create model. Then for your prediction stage (on new inaccessible dataset), write a code that will first transform any new dataset and then make a prediction. Does this make sense? $\endgroup$
    – EhsanF
    Commented Nov 8, 2017 at 13:01
  • $\begingroup$ Oops, I confused my self. You are correct Ehsan. Thank you. And Can you please tell me how does transformation basically affect predictive power ? I have data which is right skewed, i have read log transformations works best for right skewed data . I want to know how will it increase the predictive power . $\endgroup$
    – Mighty
    Commented Nov 8, 2017 at 13:17
  • 1
    $\begingroup$ You don't actually need to transform your data. Even if you are doing a Linear Regression, and if you are you would need to check that your errors are following a normal distribution. Since you're doing a classification algorithm, there is no need to get rid of the skewed data. You can bin variables using WOE to increase prediction power and robustness, but doing a log transformation on skewed data is unnecessary, specially since you're not using a Linear Regression. $\endgroup$
    – EhsanF
    Commented Nov 8, 2017 at 13:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.