I am working to preprocess a dataset where half of it is already normalized between 0 and 1. I was planning on using z-score to normalize the rest of the dataset but I was wondering if that was a bad idea. Should I use a different normalization function that keeps it between 0 and 1? Or perhaps use a z-score on the data that is normalized between 0 and 1. I know that using different normalizations is do able but I am looking for the optimal way of doing things. Also, the data set has some binary features it that makes a difference. Note: I have already one-hot encoded the categorical features. Edit: the problem is based off the the KDD Cup 1999 dataset for analyzing normal connection verse malicious connections (link to the data set here). The features are different, if you observe the dataset some of the features are "rates" so they are "normalized" between 0 and 1. Other features such as the number of failed connections are not normalized in any way. I was thinking of normalizing the number of failed connections using their z-score but I was unclear if that was a good idea based off the fact that some of the data seem to already be "normalized" since it is a rate. Let me know if you have further questions.
2
-
$\begingroup$ What problem are you trying to solve & how does scaling the data solve it? Are the original normalization constants lost? Does the second half of the data correspond to the same features as the first half, or does it represent something else -- different features, for instance? Note that z-scores aren't bounded between 0 and 1, so it's uncleear how this solves the scaling problem. $\endgroup$– Sycorax ♦Commented Mar 4, 2021 at 23:07
-
$\begingroup$ I just updated the questions so hopefully that helps $\endgroup$– LoganCommented Mar 5, 2021 at 1:46
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
Speaking specifically to the case of the KDD99 data, I've found that the features that vary over many orders of magnitude (I remember one or three go between 0 and $10^8$ or similar, for instance) are best log-transformed and then z-scored.
After the log-transformation, z-scores will give a nice scale that is amenable to gradient-based training as you'd find in a neural network. I found that without the log-transformed the data, SGD is not able to make progress, probably because the loss surface is poorly conditioned.
-
$\begingroup$ Okay thank you! Should I run the z-score on the values that range between 0 and 1? Such as the "dst_host_serror_rate" column which is already a rate between 0 and 1, but should I so a z-score so it matches my other values? $\endgroup$– LoganCommented Mar 5, 2021 at 15:33
-
$\begingroup$ Try it and see. The reason that scaling matters is to improve the performance of the gradient-based training, so if you find a method that makes the training proceed more quickly, you've succeeded. $\endgroup$– Sycorax ♦Commented Mar 5, 2021 at 15:36