0
$\begingroup$

I am doing some data analysis on a fairly large health data set of patients with diagnoses and the respective procedures received for each event. I was asked to run a multinomial logistic regression on my data.

The dataset has around 4,000 columns of attributes, of which around 3,000 are unique diagnoses. The diagnosis variables take on the value of 1 if the patient had that diagnosis and 0 if he or she did not. The remaining approximately 1,000 variables pertain to unique procedures, which also take on the value of 1 if the patient has received it, and 0 if he or she did not.

The dataset contains information on approximately 30,000 patients. I, admittedly naively, ran a the multinom function in the multinom package in R on all 4,000 variables, with the dependent variable being the very last procedure the patient has received (marked as "Final procedure" in the matrix), but R isn't able to complete the computation.

I would like some overall advice in perhaps a different package I could use for running regressions on large data sets (cannot use bigmemory however because this is on windows) or even perhaps reformatting my data.

Initially, my data set had around 50 columns, because the maximum number of diagnoses and procedures a patient had was 25 diagnoses and 25 procedures, so each column was marked as "Diagnosis X" and "Procedure x," with the corresponding element being the actual diagnosis/procedure identifier. For all the patients who did I have all 25 diagnoses/procedures (so most of them), the values in the data frame would just be NA. Now I am wondering if I could perhaps resort to using this data frame instead and have a nicer, smaller matrix to work with? The only real reason I reformatted my data set into the much larger matrix was because my grad student asked me to do so, but maybe this isn't the way to go.

$\endgroup$
  • $\begingroup$ Please clarify what you are trying to accomplish with this model, i.e. are you trying to see what previous diagnoses are associated with most recent diagnosis? Second, clarify whether you have other patient characteristics besides diagnoses and procedures among the 4,000 variables in your dataset. I'm not sure why you would fit a model with 4,000 covariates. Even if you had no missing values, the results would likely not be useful. Third, you should talk to the graduate student, as he/she is not a mind-reader and cannot know about your confusion unless you talk to him/her about it. $\endgroup$ – Marquis de Carabas Jul 21 '15 at 15:27
1
$\begingroup$

Fundamental to your issues is understanding of the methodology of health services research and clinical epidemiology. It will not be possible or meaningful to do what you originally set out to do. Instead I recommend studying the various approaches to comorbidity adjustment and risk scores, and especially to get very familiar with the various diagnosis code groupers there are. You need subject matter-guided data reduction (and not outcome-guided).

$\endgroup$
0
$\begingroup$

Then, on the purely technical side: What have you got in the "Final procedure" column? Is it "0s" and "1s" or is it numbers ranging from 1 to 1000, denoting the latest procedure each patient received. If it is 0/1, then your response (or dependent) variable is not multinomial but binomial; if it is 1-1000, then it is multinomial but it could adopt 1000 different possible values...which would require R to compute 999 different equations, each with 3,000 parameters, from only 30,000 data points, clearly insufficient. That is why, I think, R is unable to perform the computation; not lack of memory.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.