I am currently training different models (neural network, support vector machine, and XGBoost) to predict concentrations of antibiotics required to prevent growth of a bacteria from whole genome sequences.
The features for all models are the number of appearances of all substrings of length 11 (e.g. 'AGTCCGATACG' might appear 23 times in a specific sample). There are roughly 2.1 million features for 2260 samples. Using sklearn's SelectKBest() the feature size is reduced to between 100-10000 features. There are 6 possible classes (1,2,4,8,16,32) which are encoded to 0-5 and most of the data falls under the first and last classes.
Here is what the accuracy looks like against the number of features for the 3 models for a 5 fold cross validation.
The neural network has accuracy's consistent with the other 2 models until about 1700 features, after which the accuracy of the model drops off. I am hoping that I am providing enough information that someone can help answer why this is happening.
Here is what my NN looks like:
patience = 16 early_stop = EarlyStopping(monitor='loss', patience=patience, verbose=1, min_delta=0.005, mode='auto') reduce_LR = ReduceLROnPlateau(monitor='loss', factor= 0.1, patience=(patience/2), verbose = 1, min_delta=0.005,mode = 'auto', cooldown=0, min_lr=0) model = Sequential() model.add(Dense(num_feats,activation='relu',input_dim=(num_feats))) model.add(Dropout(0.50)) model.add(Dense(int(((num_feats+num_classes)/2)), activation='relu', kernel_initializer='uniform')) model.add(Dropout(0.50)) model.add(Dense(num_classes, kernel_initializer='uniform', activation='softmax')) model.compile(loss='poisson', metrics=['accuracy'], optimizer='adam') model.fit(x_train, y_train, epochs=100, verbose=1, callbacks=[early_stop, reduce_LR])
I have tried different parameters and numbers of layers to no avail. Using hyperas (keras+hyperopt) to search up to 5 hidden layers, changing dropouts, and trying different cutoffs for the early stopping and reduce_lr callbacks gave parameters that are only accurate when applied to the testing set. When using a 6th set for validation that hyperas hasn't seen, the model is no longer extendable to that dataset.
In hyperas I tried between 0-5 hidden layers, different dropouts between each layer, the number of neurons in each layer, and how soon the callbacks kicked in.
Does anyone have any insight as to why the neural network is the only model that doesn't train past a certain feature size?