2
$\begingroup$

I have a complex pipeline for predictive modeling of text, where the non-negative matrix factorization (NMF) is one part. I would like to evaluate the performance of the NMF independently of the neural network model that it is fed into afterwards. This means that I would like to evaluate the NMF in an unsupervised manner without any labels. In particular, I want to find a fitting value for the L1/L2 regularization term, alpha, in http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html.

It is important for me to optimize this regularization parameter as I want to use the NMF to remove noise in my dataset before feeding it into a classifier. What method/measure can I use to find the best performing value for alpha?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ @amoeba It's a regularization term to prevent overfitting. I've updated the post. $\endgroup$
    – pir
    Commented Mar 25, 2016 at 15:08

1 Answer 1

Reset to default
5
$\begingroup$

When dimensionality reduction techniques are part of larger pipeline, what really matters is how the reduction helps the end goal. If at all possible, I would try to see how different values of alpha affect the resulting predictions.

Since that may not always be feasible, a common way to evaluate dimensionality reduction is through reconstruction accuracy. For a simple example, one often selects the number of principal components by looking at the variance explained, which is equivalent to using the squared error of the reconstruction. There is usually no clear cut way to select the number of principal components, however, because the accuracy always increases with increasing principal components.

Similarly, decreasing alpha necessitates increases in reconstruction accuracy. A related option is to hold out a portion of the data (set to missing) and see how well the decomposition predicts the values of the held out data. It is then straight-forward to select the alpha that has the largest accuracy. For example, if the elements of your non-negative matrix are the counts of the number of times words are used in documents, you may randomly set a portion of the counts to missing and see how well the decomposition predicts the missing values for different values of alpha, choosing the alpha with the highest accuracy.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Interesting. Hadn't thought about the last option for evaluating NMF. How many counts would you set missing: 1%, 5%, 10%? And just at random throughout the matrix? $\endgroup$
    – pir
    Commented Mar 25, 2016 at 14:58
  • $\begingroup$ +1. My answer here stats.stackexchange.com/questions/111205 is related to your third paragraph (cc to @pir). $\endgroup$
    – amoeba
    Commented Mar 25, 2016 at 15:05
  • $\begingroup$ The percent to set missing is pretty subjective. I usually do around 20% or 25%. If you set too little missing, the validation will be noisy. If you set too much missing, you may not be training the model to its full capacity. You could also do this repeatedly or do cross validation. $\endgroup$
    – Andrew
    Commented Mar 26, 2016 at 22:21
  • $\begingroup$ @Andrew: What software/library/function are you using to perform NMF in the presence of missing values? Is it straightforward? $\endgroup$
    – amoeba
    Commented Mar 26, 2016 at 22:26
  • $\begingroup$ I have never used NMF with missing data. This post showed up when I searched. The nmf_update.lsnmf function seems to allow weights, which you can set to 0 for missing entries. $\endgroup$
    – Andrew
    Commented Mar 29, 2016 at 16:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.