For couple of weeks now I've been dealing with a classification problem involving a sparse dataset. To be more specific, for each input $x^{(i)}$, knowing that I have 1000 features, I've only 5 to 10 values of $k$ such as $x^{(i)}_{k} \not= 0$.

Up until now, I've tried stochastic gradient descent (SGD), momentum SGD, adagrad and adadelta. Adadelta seems to be performing the most; I think it's more down to the fact that there's no learning rate. For the previous optimizations algorithms, I struggled with the choice of $\alpha$ and I couldn't find one value that was giving good enough results for various training sets.

Anyway, this pushed me to wonder if there are specific optimization algorithms that are known to have good results with sparse data? What would you suggest? Also, I have to add that my data is unbalanced (positive class represents 8% of the dataset).

  • $\begingroup$ Naive Bayes, perhaps? It's known to work well with sparse data (the sklearn versions take sparse matrices, even), and is suitable for unbalanced datasets. YMMV. $\endgroup$ – Ami Tavory Jul 6 '17 at 5:31
  • $\begingroup$ Naive Bayes would be a learning model / classifier. In my case, I've already chosen Factorization Machines as the model to work with and I'm more worried about the optimization algorithm to learn its parameters. $\endgroup$ – mlx Jul 6 '17 at 16:23
  • 2
    $\begingroup$ You need to specify that you're using concrete algorithm - there is no such mention in post. $\endgroup$ – Jakub Bartczuk Feb 7 '18 at 12:34
  • 4
    $\begingroup$ Optimization algorithms optimize functions, they are not directly applied to the data, so you need to tell us more about the actual function you are optimizing. $\endgroup$ – Tim Apr 18 '18 at 10:26
  • $\begingroup$ I'm voting to close because the question is under-specified: optimization tasks are characterized by what function you're optimizing. Data sparsity may play a role in that function, but it is at best ancillary to the main task of optimization. $\endgroup$ – Sycorax Jul 21 '20 at 1:13

I'd suggest Lightning, because

(By the way, it doesn't add a column of ones for intercepts; do that yourself with
from sklearn.preprocessing.data import add_dummy_feature .)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.