I have a complex pipeline for predictive modeling of text, where the non-negative matrix factorization (NMF) is one part. I would like to evaluate the performance of the NMF independently of the neural network model that it is fed into afterwards. This means that I would like to evaluate the NMF in an unsupervised manner without any labels. In particular, I want to find a fitting value for the L1/L2 regularization term, alpha, in http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html.

It is important for me to optimize this regularization parameter as I want to use the NMF to remove noise in my dataset before feeding it into a classifier. What method/measure can I use to find the best performing value for alpha?

  • 1
    $\begingroup$ @amoeba It's a regularization term to prevent overfitting. I've updated the post. $\endgroup$
    – pir
    Mar 25 '16 at 15:08

When dimensionality reduction techniques are part of larger pipeline, what really matters is how the reduction helps the end goal. If at all possible, I would try to see how different values of alpha affect the resulting predictions.

Since that may not always be feasible, a common way to evaluate dimensionality reduction is through reconstruction accuracy. For a simple example, one often selects the number of principal components by looking at the variance explained, which is equivalent to using the squared error of the reconstruction. There is usually no clear cut way to select the number of principal components, however, because the accuracy always increases with increasing principal components.

Similarly, decreasing alpha necessitates increases in reconstruction accuracy. A related option is to hold out a portion of the data (set to missing) and see how well the decomposition predicts the values of the held out data. It is then straight-forward to select the alpha that has the largest accuracy. For example, if the elements of your non-negative matrix are the counts of the number of times words are used in documents, you may randomly set a portion of the counts to missing and see how well the decomposition predicts the missing values for different values of alpha, choosing the alpha with the highest accuracy.

  • $\begingroup$ Interesting. Hadn't thought about the last option for evaluating NMF. How many counts would you set missing: 1%, 5%, 10%? And just at random throughout the matrix? $\endgroup$
    – pir
    Mar 25 '16 at 14:58
  • $\begingroup$ +1. My answer here stats.stackexchange.com/questions/111205 is related to your third paragraph (cc to @pir). $\endgroup$
    – amoeba
    Mar 25 '16 at 15:05
  • $\begingroup$ The percent to set missing is pretty subjective. I usually do around 20% or 25%. If you set too little missing, the validation will be noisy. If you set too much missing, you may not be training the model to its full capacity. You could also do this repeatedly or do cross validation. $\endgroup$
    – Andrew
    Mar 26 '16 at 22:21
  • $\begingroup$ @Andrew: What software/library/function are you using to perform NMF in the presence of missing values? Is it straightforward? $\endgroup$
    – amoeba
    Mar 26 '16 at 22:26
  • $\begingroup$ I have never used NMF with missing data. This post showed up when I searched. The nmf_update.lsnmf function seems to allow weights, which you can set to 0 for missing entries. $\endgroup$
    – Andrew
    Mar 29 '16 at 16:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.