I'm working on classifying UNSW-NB15 dataset into 2 categories (benign - malware) using neural network. The full dataset include about 2.000.000 benign samples and 300000 malware samples. I'm assuming that is the approximate proportion of real-life samples. So should I just randomly sample training/validating/testing set with the same benign/malware ratio but with different size (e.x 60 %, 20%, 20% respectively) or should I do it another way?
Use stratified random sampling to maintain the benign/malware ratio in your sets. More advanced stratified sampling may be warranted, but you need learn the topic. For example, if you know of a predictor that has an uncommon level which is highly significant, you can include that in the call the stratified sampling function. You can also stratify on all the levels of all predictors, where appropriate.
From the Encyclopedia of Research Design:https://methods.sagepub.com/reference/encyc-of-research-design/n445.xml
Stratified random sampling (usually referred to simply as stratified sampling) is a type of probability sampling that allows researchers to improve precision (reduce error) relative to simple random sampling (SRS). The population is divided into non-overlapping groups, or strata, along a relevant dimension such as gender, ethnicity, political affiliation, and so on. The researcher then collects a random sample of population members from within each stratum. This technique ensures that observations from all relevant strata are included in the sample.