How to Reduce Number of Variables Before Running Random Forrest or XGBoost I've simplified the problem I'm working on for this post, so that the focus is on the issue I'm having.
I'm trying to predict if a patient will be diagnosed with arthritis in 2019, based on the ICD-10 diagnosis codes found on claims in 2018. My strategy was to build a model off of a training set based on 2017 claims, with a target variable that indicates if the patient had arthritis in 2018. I would then score patients based on their 2018 claims.
Below is a screenshot of the way I wanted to structure my data, with some made up diagnosis data.

Patient ID  Arthritis_Ind   Age Gender  Diag_A00_Ind    Diag_A000_Ind   Diag_A001_Ind   […]
100000001   0   75  F   0   0   1   […]
100000002   0   60  M   0   0   0   […]
100000003   1   71  F   0   1   0   […]
100000004   0   80  M   1   0   0   […]
100000005   0   91  F   0   0   0   […]
[…] […] […] […] […] […] […] […]
The columns in my table are as follows:
Patient ID: Uniquely identifies a patient. There will be one row for one patient.
Arthritis_Ind: Indicates of the patient was diagnosed with arthritis in 2018, with a 1 or 0.
Age: Patient age (Integer).
Gender: Patient gender (M or F).
Diag_XX_Ind: An indicator variable, 1 or 0, that indicates that a member received a diagnosis at sometime in 2017, where XX represents an ICD-10 diagnosis code.
The problem is, there are 18,000 ICD-10 diagnosis codes, and I have data on hundreds of thousands of patients. I'm running into memory issues creating a table with this many columns and creating a random forest or xgboost model off of it.
I strongly suspect that, at most, only a few dozen ICD-10 diagnosis codes will predict if a patient has arthritis. Any advice as to how I can reduce the list of 18,000 potential ICD-10 diagnosis codes down to a more manageable number, before creating a random forest or xgboost model? Are there any methods I could use to reduce the number of variables before hand? Should I restructure my data in some way? Any advice would be much appreciated. Thanks!
 A: I was able to come up with a solution to this problem, and wanted to post here in case anyone else found it helpful. Comments by usεr11852 about sparse encoding helped point me in the right direction.
My strategy was as follows.
1) Restructure my SQL pull in a long format, to avoid the column issue I was facing in SQL. See example below:
Patient_ID,Var,Val
100000001,_Arthritis_Ind,0
100000001,Age,75
100000001,Diag_A001_Ind,1
100000002,_Arthritis_Ind,0
100000002,Age,60
100000003,_Arthritis_Ind,0
100000003,Age,71
100000003,Diag_A000_Ind,1
etc...
2) Use the sparseMatrix function in the R package Matrix to reshape the data from long into wide format. Because the matrix is extremely sparse, I didn't run into memory issues. The method is described at this link:
https://www.r-bloggers.com/casting-a-wide-and-sparse-matrix-in-r/
3) Run xgboost, with _Arthritis_Ind as my target variable, which ran without issues.
A: I would start with the most simple possible solution, and only if that doesn't work I'd start to look into more complicated options.
The first place to start when you wish to reduce the number of variables in your data is principal components analysis (PCA). This looks for new features that are linear combinations of your old features, and summarize your dataset, minimizing the information lost in the process. If your data is sparse (you said that the ICD-10 codes are just ones and zeros), the chances are that just a few PCA features (called principal components) will summarize your dataset well. 
PCA is implemented out-of-the box in scikit-learn. You can import it with sklearn.decomposition.PCA, train it with pca = PCA(n_components=q).fit(X_train) and use it on your train or test data with pca.transform(X_train_or_test).
I would keep the rest of your features, run PCA just on the ICD-10 codes and find a few tens or hundreds of principal components for the ICD-10 codes. In scikit-learn, you can choose the number of principal components by looking at the pca.explained_variance_ratio_. An accumulated explained variance ratio of 0 explains nothing of your dataset, and a ratio of 1 explains everything.
Good luck!
(Gist of PCA)
If $\mathbf{X}$ is your data matrix (called the design matrix, with $N$ samples along the row dimension and $p$ features along the column dimension), PCA finds the $q$ principal components $\mathbf{v}$, $q \leq p$, that most minimize the difference between $\mathbf{X}$ and $\mathbf{Xvv}^\text{T}$.
$$
\DeclareMathOperator*{\argmin}{arg\,min}
    \mathbf{v^*} = \argmin_\mathbf{v} \;\;\lVert \mathbf{X} - \mathbf{Xvv}^\text{T} \rVert^2,
$$
where $\lVert \cdot \rVert$ is the Frobenius norm (essentially the same as the $L2$ norm but for matrices). Notice that if $\mathbf{v}$ is an unit vector that specifies a direction, $\mathbf{Xvv}^\text{T}$ is simply the reconstruction of $\mathbf{X}$ along that direction. So PCA is finding the directions with the most information!
(End of gist of PCA)
