I tried to do a OLS linear regression and got the following error message the end of the output:
"2 The smallest eigenvalue is 1e-25. This might indicate that there are strong multicollinearity problems or that the design matrix is singular."
I thought regression was supposed to solve/account for multicollinearity. Should I do PCA before regression and try this again?
How do I solve this error message? Is it even significant?
Input Data:
EDIT: Took advice from commenter.
Unless your data has more than two genders, including a constant and both male and female as indicators will be collinear.
Let $m_i$ be an indicator for male and let $f_i$ be an indicator for female. Observe that for any individual $i$
$$m_i + f_i = 1$$
hence if you also include a constant, your data is collinear.
If you have indicators for various categories (eg. month of the year), you should leave an indicator for one of the categories out of the regression. Coefficients on the remaining indicators will then be an estimate of the categories' mean relative to the left out category.
Think through the logic of your problem. You can have some subtle stuff such as age, years of schooling, and a constant being collinear if everyone starts school at the same age and doesn't drop out.
If you're doing machine learning with tons of variables, you may want to explore regularization.
Some further ideas and examples are explored here.
First of all, you should to be sure that you have multicollinearity. Check correlations between variables and use the VIF factor.
Then, if you want to solve multicollinearity reducing number of variables with a transformation, you could use a multidimensional scaling using some distance that remove redundancies. PCA is useful when you have continuous variables, but I'm not sure that was good when your data is binary, so maybe you need to use MDS with binary ED or CorEx.
