My dataset has 115 points and I am cleaning/processing the data in python. My understanding is that having a symmetric normal distribution ensures that when the model is trained, it will not have a bias on one side of the data. It also means there are no outliers on one side of the data.

As a result I am transforming my data (log, power, exp) to get as close to a normal distribution and then checking the skewness using pandas skew() method. I have performed 2 transformations:

Transformation 1 (image 1 and 2) - This results in the data getting very close to normal distribution but the general trend from scatter graph could be stronger.

Transformation 2 (image 3 and 4) - The data is positively skewed, however the general trend from the scatter graph is very strong.

Could someone clarify if skewness should be prioritised over the general trend of the dataset. There are machine learning algorithms that can determine a non-linear relationship (e.g. support vector regression). Will skewness also have an impact of tree based models (e.g. decision tree)?

enter image description here

enter image description here

enter image description here

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.