PCA, regression and/or LASSO?

Although there is lots of information on all three subjects, I could not find information on how to solve my specific issue. Most likely that is because I am not an expert in statistics and might have overlooked something. If someone knows of a link that deals with my specific question I will be most happy. If there is no such thing, maybe you can explain what I need to do.

I have a 4000 x 10 matrix (10 columns by 4000 rows). Column 1 is the dependent variable, all other columns are explanatory variables. Columns 2:6 are all categorical variables ( and contain strings as 'on' versus 'off', and 'South' versus 'West' versus 'North' versus 'East', etc.), and columns 7:10 are all numeric AND are correlated (to various degrees) to one another. There is no a priori knowledge of any correlations among vectors 2:6 or between any vectors 2:6 versus any vectors 7:10.

I want to determine a linear regression model. I could use PCA to address the correlated variables but what about the categorical variables? I can apply lasso, but what about the correlated variables? And can I address lasso's issue with categorical variables by using group lasso?

What would be the correct procedure to obtain a best linear regression model for this dataset? I use matlab but I am not after any matlab coding but I want to understand the process steps I need to undertake.

• You could use PCA on the numerical variables, then attach the PCA results to the categorical vars and run linear regression. – Rahul Jul 19 '17 at 1:48
• Thanks Rahul. Applying (standard) linear regression can be done on categorical data and correlated data. However, if I want to use lasso as regression I will have issues both with the categorical data and correlated data. I am interested in finding out whether lasso will help me with the anova analysis on which model representation is best. – Frank Drost Jul 19 '17 at 2:08
• Rahul mentioned doing PCA first; after that you could use lasso just as well as a normal linear regression. Aside from being an initial step to take out components with low explained variance (if you choose to do so before running lasso), it also uncorrelates your numeric variables. – Kevin Jul 19 '17 at 3:36
• @FrankDrost I still don't see an issue. PCA(numericals) + Categoricals -> set of new predictors. Lasso can handle categoricals just fine. (Creation of dummy variables and all that...) – Rahul Jul 19 '17 at 4:29
• I just found out that Matlab's lasso doesn't deal with categorical data (au.mathworks.com/help/stats/…). I have to see what I can do next. – Frank Drost Jul 19 '17 at 5:03