1
$\begingroup$

I downloaded the 2013 dataset from Kyoto University.

My question is this: how will I know if I should [feature] scale or standardize my data?.

Thank you in advance!

UPDATE 07/13/2017a

This is the script I wrote for getting the data summary using pandas.

UPDATE 07/13/2017b

Here's the data summary, with description of index [22]:

                 0             2             3             4             5   \
count  4.656124e+06  4.656124e+06  4.656124e+06  4.656124e+06  4.656124e+06   
mean   5.629711e+00  7.620859e+03  8.519262e+03  1.819146e+00  2.604244e-01   
std    1.747110e+02  3.446973e+06  3.346247e+06  8.298907e+00  4.228922e-01   
min    0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00   
25%    0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00   
50%    2.866777e+00  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00   
75%    3.419435e+00  1.280000e+02  2.090000e+02  1.000000e+00  5.000000e-01   
max    8.097288e+04  2.133443e+09  2.116371e+09  1.000000e+02  1.000000e+00   

             6             7             8             9             10  \
count  4.656124e+06  4.656124e+06  4.656124e+06  4.656124e+06  4.656124e+06   
mean   5.863585e-02  4.008782e-01  1.056284e+01  2.819324e+01  3.256177e-02   
std    2.289534e-01  4.238607e-01  2.233580e+01  2.824031e+01  1.718919e-01   
min    0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00   
25%    0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00  0.000000e+00   
50%    0.000000e+00  3.300000e-01  0.000000e+00  3.000000e+01  0.000000e+00   
75%    0.000000e+00  9.500000e-01  5.000000e+00  5.000000e+01  0.000000e+00   
max    1.000000e+00  1.000000e+00  1.000000e+02  1.000000e+02  1.000000e+00   

             11            12            17            19            21  \
count  4.656124e+06  4.656124e+06  4.656124e+06  4.656124e+06  4.656124e+06   
mean   1.405558e-01  2.122936e-01 -3.552655e-01  2.292569e+04  1.648552e+03   
std    3.280353e-01  3.864984e-01  9.440645e-01  2.250753e+04  6.820971e+03   
min    0.000000e+00  0.000000e+00 -2.000000e+00  0.000000e+00  0.000000e+00   
25%    0.000000e+00  0.000000e+00 -1.000000e+00  3.028000e+03  2.500000e+01   
50%    0.000000e+00  0.000000e+00 -1.000000e+00  6.000000e+03  8.000000e+01   
75%    0.000000e+00  0.000000e+00  1.000000e+00  4.522200e+04  4.450000e+02   
max    1.000000e+00  1.000000e+00  1.000000e+00  6.553500e+04  6.553500e+04   

             22  
count  4.656124e+06  
mean   1.216933e+01  
std    7.080260e+00  
min    0.000000e+00  
25%    5.916667e+00  
50%    1.228806e+01  
75%    1.844000e+01  
max    2.399944e+01

UPDATE 07/19/2017

I finished writing the final script for standardizing my dataset, and here it is.

$\endgroup$
0
$\begingroup$

Standardization (or normalization using Student's t statistics, to use a longer name) is a good practice if you don't want your machine learning algorithm to improperly assign a larger relevance to the variable that has the largest variability in absolute terms, and sometimes just for easier interpretation of the machine learning algorithm parameters. That being said, some algorithms strictly require standardization (es Principal Component Analysis) and some other do not at all (Random Forests).

In order to reply to your problem, you should provide some summary of data to understand the numerical range. If you are using python+pandas, that would be df.describe(), in R, that would be summary(df). You should standardize the variables that are continuous, or quasi continuous (integers over a wide range). In your case, from what I see you should standardize the duration, bytes, counts and error rates.

You are free to normalize the same variables instead of standardizing them, but it would not make sense to standardize some and normalize others.

$\endgroup$
  • $\begingroup$ A novice question -- so, it's okay to standardize just those that you mentioned, and leave the other features to normalization? In addition, should I normalize after after standardizing those features? $\endgroup$ – afagarap Jul 12 '17 at 9:35
  • $\begingroup$ Perhaps confusion here is just about naming convention. What I am suggesting is to "normalize according to standard score" as in en.wikipedia.org/wiki/Normalization_(statistics). In other words, subtract the mean and divide by standard deviation. I don't see how you could do normalization using any kind of score (at least the ones I know) when the variable is a MAC address - or any categorical variable for that matter. $\endgroup$ – famargar Jul 12 '17 at 9:55
  • $\begingroup$ Oh, sorry. I forgot that for the experiment, I will be disregarding the IP Address features first. I'll just get back on it if I get to read a method on transforming their values as well. $\endgroup$ – afagarap Jul 12 '17 at 12:32
  • $\begingroup$ In regards with the naming convention, yeah. I got that a bit wrong. My question should have been, for other features, should I just do feature scaling instead of standardizing? $\endgroup$ – afagarap Jul 12 '17 at 12:33
  • $\begingroup$ I don't think your word usage is wrong. I used the word standardize to mean normalize according to standard score (or better using Student's t statistics, given that I assume you don't know mean and standard deviation, nor if distribution is gaussian), and you used normalize to mean normalize using unity-based normalization or simply feature scaling. Regardless, using different normalizations metrics in the same data would not make sense to me. And I prefer using Student's t statistic as it uses the information on how observations spread in your data through the standard error. $\endgroup$ – famargar Jul 12 '17 at 12:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.