0
$\begingroup$

The domain I'm operating in is Healthcare. So I have a lot of variables like visits to the doctor, prescribed drugs etc. - each one has a time dimension (5-6 years) and a "measure" dimension like count, standard deviation, average. Basically I have a LOT of columns which typically look like the following (e.g. for visits):

"visits_pediatrician_2012_avg", "visits_pediatrician_2013_avg",..., "visits_pediatrician_2016_std" - and then the next doctor specialization comes in.

I hope you get the picture. There are approximately 30k variables. It is important to note that not all of them sit in the same file, otherwise I could just reshape everything (and get a different problem of a lot of observations). I have topic files.

Another important piece of information regarding my problem is that the goal of this project is inference and explanation rather than prediction.

My question is - how can I start analyzing the variables and sorting out which ones are important to me and which ones are not? I read here that some people suggested LASSO or PCA. My problems with those are:

  1. Yes, LASSO is giving me sort of a feature selection eventually, but I totally miss any correlative relationships between 2 or more variables. It might be that x_1 alone gets dropped out by LASSO, but there is important information at the intersection between x_1 and x_2 that I miss.

  2. PCA will help me with dimension reduction, but then later at the modeling phase, I'll have "cooked" variables which are basically some sort of linear combinations of my original variables. If, for example, my model is logistic regression I wont be able to explain the results because the coefficients I'll get will be assigned to these "cooked" variables which are hard to interpret.

What can be done then to reduce the dimensionality but also to retain explanation ability and not loosing some correlative relationships among two or several more variables in the data?

$\endgroup$
1
  • $\begingroup$ It might help if you could say more about the object of your "inference and explanation." Usually, knowledge of the subject matter is the best key to choosing variables with which to start. $\endgroup$ – EdM Nov 5 '17 at 21:19
0
$\begingroup$

Have you looked at factor analysis, or principal component analysis? Roughly speaking, the goal of both is to say "given a whole bunch of variables, which ones are correlated in such a way that, if I lump them together in a proper weighting, I get a new variable which is the most powerful - then, after exploiting that variable, what is the next synthetic variable I can build that has the next most power? then...."

$\endgroup$
1
  • $\begingroup$ PCA = Principal Component Analysis. I've written about it in the post... $\endgroup$ – Corel Nov 5 '17 at 21:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.