How to build a predictive model when more levels of a categorical predictor are possible than appear in the training data

Question

I have created an Artificial Neural Network with 4 categorical features and a binary outcome either 1 for suspicious or 0 for non-suspicious:

  ParentPath                                  ParentExe
0   C:\Program Files (x86)\Wireless AutoSwitch  wrlssw.exe
1   C:\Program Files (x86)\Wireless AutoSwitch  WrlsAutoSW.exs
2   C:\Program Files (x86)\Wireless AutoSwitch  WrlsAutoSW.exs
3   C:\Windows\System32                         svchost.exe
4   C:\Program Files (x86)\Wireless AutoSwitch  WrlsAutoSW.exs

ChildPath                                   ChildExe    Suspicious
C:\Windows\System32                         conhost.exe  0
C:\Program Files (x86)\Wireless AutoSwitch  wrlssw.exe   0 
C:\Program Files (x86)\Wireless AutoSwitch  wrlssw.exe   0
C:\Program Files\Common Files               OfficeC2RClient.exe  0
C:\Program Files (x86)\Wireless AutoSwitch  wrlssw.exe  1
C:\Program Files (x86)\Wireless AutoSwitch  wrlssw.exe  0

I have used sklearn for label encoding and one hot encoding on the data:

#Import the dataset
X = DBF2.iloc[:, 0:4].values
#X = DBF2[['ParentProcess', 'ChildProcess']]
y = DBF2.iloc[:, 4].values#.ravel()

#Encoding categorical data
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
#Label Encode Parent Path
labelencoder_X_1 = LabelEncoder()
X[:, 0] = labelencoder_X_1.fit_transform(X[:, 0])
#Label Encode Parent Exe
labelencoder_X_2 = LabelEncoder()
X[:, 1] = labelencoder_X_2.fit_transform(X[:, 1])
#Label Encode Child Path
labelencoder_X_3 = LabelEncoder()
X[:, 2] = labelencoder_X_3.fit_transform(X[:, 2])
#Label Encode Child Exe
labelencoder_X_4 = LabelEncoder()
X[:, 3] = labelencoder_X_4.fit_transform(X[:, 3])

#Create dummy variables
onehotencoder = OneHotEncoder(categorical_features = [0,1,2,3])
X = onehotencoder.fit_transform(X)

I have split the data into a training and test set and run it on my gpu box with a nvidia 1080. I have tuned the hyperparameters and am now ready to use the model that is trained in a production environment with one test sample being tested at a time. Lets say I just want to test one sample:

   ParentPath            ParentExe     ChildPath           ChildExe
0  C:\Windows\Malicious  badscipt.exe  C:\Windows\System   cmd.exe  

The issue that I am running into is the training set has seen the ChildPath "C:\Windows\System" and the ChildExe "cmd.exe" which are normal, but the training set has not seen the ParentPath "C:\Windows\Malicous" or ParentExe "badscipt.exe" so these have not been label or one hot encoded. My big question is how to build a predictive model when not all of the categorical variables (in this case, file paths and file names) can be exhaustively enumerated?

I have seen examples using feature hashing but I'm not sure how to apply that or if that would even solve this problem. Any help or pointers would be greatly appreciated.

Answer 1

Categorical features that can't be fully enumerated are failure-prone

The challenge that you've discovered is a natural consequence of how you've organized your research project: your model has no generalizable information about new file paths or new names of .exe files.

This theme is very common -- suppose you're trying to predict credit default using a unique identifier such as a person's social security number. You'll find that you can do really well on your training data by assigning a risk to each SSN. However, when you have new SSNs which are not in the training data, the model has no relevant information to work with.

Or, we can extend this analogy to the malware context and just have a lookup table and declare some file as malware when the sha256 hash of the file matches a known malware sample: clearly this will have problems whenever (1) someone makes a new malware file that doesn't have a sha256 matching an existing known malware sample or (2) someone makes a malware file that has the same hash as a known clean sample.

The conceptual model is also a security flaw

From a security standpoint, there's a conceptual flaw with this approach. Whenever a malware file is located in a directory which your model thinks is clean, with a name that the model thinks is clean, the malware will evade detection.

Instead, use generalizable features for static analysis

A more robust approach would take various measurements from a corpus of .exe files and attempt to learn a model based on those features. This is the most common approach for modern machine learning applications applied to malware detection. An open-source example of such a feature extraction engine is EMBER, by Hyrum S. Anderson & Phil Roth. The EMBER paper outlines a number of generalizable features for malware detection in PE files.

Of course if you're representing software files as fixed-length feature vectors, then you can use any standard machine learning model for tabular data (logistic regression, svm, random-forest, xgboost, etc.), not solely neural networks.

• Hyrum S. Anderson & Phil Roth. EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models

A less-common approach is to "eat the whole .exe" by having the a neural network ingest the entire .exe binary and make decisions based on the binary sequence, or byte-encoded sequence. Usually this involves a combination of recurrent and convolutional network structures.

• Edward Raff, Jon Barker, Jared Sylvester, Robert Brandon, Bryan Catanzaro, Charles Nicholas, "Malware Detection by Eating a Whole EXE"
• Scott Coull, Christopher Gardner, "What are Deep Neural Networks Learning About Malware?"

Right now, hand-crafted features for malware detection are state-of-the-art. However, I wouldn't be surprised if some clever researchers can come up with a combination of a nice architecture and data augmentation strategy which can make a feature-free neural network out-perform hand-crafted feature vectors.

---

More generically, the advice at The suggestions at Encoding of categorical variables with high cardinality and Principled way of collapsing categorical variables with many levels? may also be helpful.