I am trying to create a classification model. While pre-processing the data. I look at the variance in each column. This is the amount of variance in each column. I am confused on which all columns should I log transform before modelling. How much variance is acceptable? Could somebody please shed s

Temparature     2.318567e-01
HR              4.747868e+02
SpO2            1.179291e+01
SBP             6.263887e+02
MAP             2.905884e+02
RR              2.794205e+01
FiO2            9.061920e+00
PaO2            1.327011e+03
PaCO2           7.466527e+01
pH              4.851681e-03
A.a.gradient    0.000000e+00
HCO3            1.358290e+01
Hb              5.337076e+00
TLC             6.326940e+07
Platelets       1.062145e+10
K               3.332203e-01
Na              4.429681e+01
Serum.Cr        1.897277e+00
Blood.Urea      7.321509e+02
Bili            3.352918e+00
Urine.output    5.157271e+05
Lactate         3.795719e+00
INR             5.362644e-01
dtype: float64

  • 1
    $\begingroup$ Why do you believe that there is a maximum amount of acceptable variance? $\endgroup$ – Stephan Kolassa Mar 26 at 12:52
  • $\begingroup$ I am taking this ML course and was told in a class of log transformation of variables if the variance is too high compared to the other variables. I am not sure on how to decide on when to apply log transform. $\endgroup$ – Dileep Unnikrishnan Mar 26 at 13:22
  • $\begingroup$ That sounds like advice that is safely ignorable, or at least begging of a follow up "why?". $\endgroup$ – Matthew Drury Mar 26 at 14:28
  • $\begingroup$ I can think of some situations where you might want to re-scale the variables to be meaningful, i.e. you might transform something like Gross Domestic Product from raw dollars into billions or trillions of dollars. Sometimes, vastly different scales can cause convergence trouble. However, log transforming now mean you assume the variable has a non-linear effect, and this may not be what you want to do. I, too, have never heard that you should log transform simply because the variance is over some amount. $\endgroup$ – Weiwen Ng Mar 26 at 17:07
  • $\begingroup$ Side note: is the variance of heart rate really 475 or so, and the variance of temperature really 0.232? Those look like potential data errors. $\endgroup$ – Weiwen Ng Mar 26 at 17:08

This answer argues that, "The only time you would be justified or correct in taking the Log of $Y$ is when it can be proven that the Variance of $Y$ is proportional to the Expected Value of $Y^2$".

If there is high variance but it is inappropriate to log transform, one way to deal with some columns having higher variance than others is to put them on equal footing by performing input normalization

This ensures that all your columns have a variance of 1.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.