13
$\begingroup$

Table 18.1 in the Elements of Statistical Learning summarizes the performance of several classifiers on a 14 class data set. I am comparing a new algorithm with the lasso and elastic net for such multiclass classification problems.

Using glmnet version 1.5.3 (R 2.13.0) I am not able to reproduce point 7. (the $L_1$-penalized multinomial) in the table, where the number of genes used is reported to be 269 and the test error is 13 out of 54. The data used is this 14-cancer microarray data set. Whatever I have tried, I get a best performing model using in the neighborhood of 170-180 genes with a test error of 16 out of 54.

Note that in the beginning of Section 18.3, on page 654, some preprocessing of the data is described.

I have contacted the authors -- so far without response -- and I ask if anybody can either confirm that there is a problem in reproducing the table or provide a solution on how to reproduce the table.

$\endgroup$
  • 1
    $\begingroup$ glmnet has been undergoing quite a bit of change recently and has had some problems with numerics in the past. Is it possibly due to this? How long since you contacted the authors? I see the current version is 1.7 and was uploaded to CRAN only about a week ago. $\endgroup$ – cardinal Jun 27 '11 at 14:42
  • $\begingroup$ @cardinal, it was about four weeks since I did the last experiments with glmnet, but we also have a different implementation that produces similar results not consistent with the table in ESL. The table is definitely older, so my guess is that the table is not correct, but it would be nice to know for sure. $\endgroup$ – NRH Jun 27 '11 at 17:52
  • $\begingroup$ I very briefly skimmed those sections and one question that came up in my mind was how the cross validation was done to pick the shrinkage parameter in (18.19) on page 661 (third printing). Any idea? Maybe I missed it or it's described elsewhere? That seems like a likely place where your attempts to recreate their analysis could be sensitive to differences in the approach. $\endgroup$ – cardinal Jun 30 '11 at 15:06
  • $\begingroup$ @cardinal, first thanks for taking an interest in this. It is correct that CV can make a difference, but the authors actually have the subsets (indices) used for CV on the web page together with the data. Anyway, CV is only used for selecting the optimal penalty parameter lambda, then the whole training data set is used to fit the model, which is then assessed on the test data. Hence, even if the CV step selects a different lambda, that lambda is on the solution path for the training data, and we can't find it ... $\endgroup$ – NRH Jul 3 '11 at 21:28
2
$\begingroup$

have you checked the R package of the book? it contains all the datasets, function and most of the scripts used in there...

$\endgroup$
  • 7
    $\begingroup$ nice try. Yes, I have checked the package, but claiming that it contains all data, functions and most scripts is an exaggeration. It is not complete, and it does not contain the data set in question. $\endgroup$ – NRH Jul 26 '11 at 21:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.