How to analyse a data set with more than 500 variables I have a data set with 1000 entities. For each entity, more than 500 dichotomous variables (0,1) are recorded. I have no idea how to analyse this data or how to get an overview.
My problem is not that any analysis I attempt takes too long to run on the system, but that the result is equally confusing.
My question is whether there are statistical methods to aggregate this data. Or to get an overview, e.g. of the correlation between variables.
Or in general, if there are special statistical methods to handle data sets with such a lot of variables in a way that the result is a bit more compact.
I don't even know how to start and would appreciate any suggestions.
 A: Asking "how do I analyze this dataset" without reference to a research question is like asking "what do I do with this pile of wood." Well, you could turn it into a house, or a desk, or a chair, or an abstract sculpture. It depends on what you need, and there is no way for anyone to give you advice on what to do unless you tell them what you are trying to do.. That being said, once you have decided you want to build a chair, then there are some clear guidelines and best practices to help you build a GOOD chair (and avoid making a bad one).
Similarly, Data analysis is a process of using various mathematical tools (analogous to hammers and screwdrivers) to answer questions using data (the "raw material"). It doesn't matter if your dataset has 2 variables or 500 or a million, you can't coherently talk about "analyzing" it except in reference to a specific question or set of questions. Once you have a question then you can think about which variables in the dataset, and what analytic tools, could help you to answer that question.
So for example, a really common kind of question in data analysis involves trying to figure out whether a specific "dependent variable" is in some way associated with (or caused by?) one or more independent variables, after accounting for other confounding variables. This sort of question is often analyzed using tools like regression analysis, but setting up the model depends on what particular dependent and independent variables you are interested in. Does participating in the program make people more likely to get a job? Do Black patients get the same care as White patients? Are different kinds of diets effective at loosing weight?
Another type of question you might ask is whether a bunch of different questions all measure one or more underlying latent constructs. Do these 10 questions about "depression" all tap some underlying construct of "depression" and if so is there a way to aggregate this data together to create a single measure of "depression?" This kind of question of often answered with tools like factor analysis or principle components analysis.
But again, none of this has any meaning in the abstract. Data analysis is about asking and answering questions. Without a question, the data is just a pile of wood.
