0
$\begingroup$

I downloaded a dataset for intrusion detection. It's from the honeypot systems of Kyoto University (2013 dataset). I'll be using the dataset for training a neural network. My problem is how to process the dataset. Am I going to use vector space model or what?

A sample from the dataset:

0.000000,other,0,0,0,0.00,0.00,0.00,1,37,1.00,0.00,0.00,RSTOS0,0,0,0,1,fd75:41fb:cf76:971a:0b40:03a2:4ab7:0f81,38581,fd75:41fb:cf76:dc4c:7d2c:2705:07b2:0f45,25,00:00:00,tcp

The specifications of the features are as follows (further details):

  1. Duration
  2. Service
  3. Source bytes
  4. Destination bytes
  5. Count
  6. Same_srv_rate
  7. Serror_rate
  8. Srv_serror_rate
  9. Dst_host_count
  10. Dst_host_srv_count
  11. Dst_host_same_src_port_rate
  12. Dst_host_serror_rate
  13. Dst_host_srv_serror_rate
  14. Flag
  15. IDS_detection
  16. Malware_detection
  17. Ashula_detection
  18. Label
  19. Source_IP_Address
  20. Source_Port_Number
  21. Destination_IP_Address
  22. Destination_Port_Number
  23. Start_Time
  24. Duration
Improve this question
$\endgroup$
5
  • $\begingroup$ If it is a publicly available dataset, i doubt that you are the first one to work on it. I recommend to read papers on how others preprocessed the same dataset $\endgroup$
    – Nikolas Rieble
    Commented Jul 10, 2017 at 9:18
  • $\begingroup$ Thanks, @NikolasRieble I did read one paper, the other night. What they did was to index the symbolic features like service to [0, n-1] where n is the number of symbols. As for the integer values, they did a linear scaling to [0.0, 1.0]. What I am now having a trouble with is how they did the linear scaling. $\endgroup$
    – afagarap
    Commented Jul 11, 2017 at 5:02
  • $\begingroup$ If I understand correctly, a linear scaling here means first $x_i = x_i - min_i(x_i)$ and the $x_i = x_i / max_i(x_i)$ for all values. $\endgroup$
    – Nikolas Rieble
    Commented Jul 11, 2017 at 9:15
  • $\begingroup$ I actually defined it as a function $f(x) = \frac{(b - a) \times (x - min)}{(max - min)} + a$ where $[min, max] \rightarrow [a, b]$. $\endgroup$
    – afagarap
    Commented Jul 11, 2017 at 9:51
  • $\begingroup$ $[a, b]$ is the scale range, for my problem, it's $[0.0, 1.0]$. $\endgroup$
    – afagarap
    Commented Jul 11, 2017 at 9:52

1 Answer 1

Reset to default
0
$\begingroup$

For the categorical data like service and flag, I'll be indexing them to $[0, n-1]$ where $n$ is the number of categories in the feature.

As for the numerical data, I'll be standardizing them using the Student's t-statistic $z = \dfrac{x - \mu}{\sigma}$.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.