3
$\begingroup$

I have a classification problem, with about 10 different inputs, some boolean, some categorical (and unrelated to each other), some being a float between 0 and 1, which need to be mapped to 4 different outputs.

My problem is that the amount of data that I have is relatively limited. I have about 10,000 data points.

What would make more sense here? Boosted Trees or a Neural Network? I wonder if it makes sense to use a Neural Network at all, given that training an NN seems to require much more data.

Please note I don't want to use SVM, k-means, etc, ideally want to make one of these two methods work.

Also what parameters would you suggest? Like number of trees/leafs? Number of hidden layers? I know a lot of it boils down to experimentation, but what are good/proven starting values to get good results?

$\endgroup$

closed as too broad by Michael R. Chernick, mdewey, Carl, whuber Nov 15 '18 at 19:39

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    $\begingroup$ There's no generic answer to this question. If there's existing literature on how people have solved this problem in the past, start there and see if you can beat their best effort. Otherwise, you'll have to try things until you're satisfied with the results. $\endgroup$ – Reinstate Monica Nov 12 '18 at 21:10
  • 3
    $\begingroup$ No way to tell unless you try. Generally, I would avoid NN when you have such little training data. I'm also a little concerned with your statement about "making one of these two methods work". Feels like you may be forcing a square peg in a round hole. $\endgroup$ – Demetri Pananos Nov 13 '18 at 4:37
4
$\begingroup$

You cannot determine which machine learning algorithm and hyperparameters are ideal until you fit models based on a combination of machine learning algorithms and hyperparameters. To find out which algorithm is optimal, you'll have to try a few beyond just the two you've listed.

To do this, use a nested cross-validation approach to optimize which combination of hyperparameters to use for each machine learning technique.

  1. Split the data into training and testing sets.
  2. For each machine learning algorithm, determine potential combinations of hyperparameters by examining the specifications of each machine learning algorithm. For each combination of hyperparameters, fit models using the training data and cross-validation; and calculate the mean accuracy. Choose the model with the hyperparameter combination with the highest mean accuracy. This will be the optimal model for this machine learning algorithm.
  3. Compare the accuracy of each optimal model between the machine learning algorithms by testing them using testing sets.

Here are some good references with some examples:

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.