I am doing some data analysis on a fairly large health data set of patients with diagnoses and the respective procedures received for each event. I was asked to run a multinomial logistic regression on my data.
The dataset has around 4,000 columns of attributes, of which around 3,000 are unique diagnoses. The diagnosis variables take on the value of 1 if the patient had that diagnosis and 0 if he or she did not. The remaining approximately 1,000 variables pertain to unique procedures, which also take on the value of 1 if the patient has received it, and 0 if he or she did not.
The dataset contains information on approximately 30,000 patients. I, admittedly naively, ran a the multinom function in the multinom package in R on all 4,000 variables, with the dependent variable being the very last procedure the patient has received (marked as "Final procedure" in the matrix), but R isn't able to complete the computation.
I would like some overall advice in perhaps a different package I could use for running regressions on large data sets (cannot use bigmemory however because this is on windows) or even perhaps reformatting my data.
Initially, my data set had around 50 columns, because the maximum number of diagnoses and procedures a patient had was 25 diagnoses and 25 procedures, so each column was marked as "Diagnosis X" and "Procedure x," with the corresponding element being the actual diagnosis/procedure identifier. For all the patients who did I have all 25 diagnoses/procedures (so most of them), the values in the data frame would just be NA. Now I am wondering if I could perhaps resort to using this data frame instead and have a nicer, smaller matrix to work with? The only real reason I reformatted my data set into the much larger matrix was because my grad student asked me to do so, but maybe this isn't the way to go.